AI Is Great, But Not Everyone Is Ready to Use It Yet

Access isn’t the same as knowing how

I’ve spent a lot of this year experimenting with AI, and I think it’s great. It writes code, drafts documents, sketches screens and answers questions in fields I don’t work in. Some days it feels like I have a small team sitting in my terminal.

What I keep noticing is how different the results are from one person to the next. Two people can use the same tool, on the same subscription, on the same kind of task, and get very different outcomes. One ships something useful. The other spends the afternoon going in circles with a confident assistant.

That shouldn’t surprise anyone who has been in this industry for a while. Every technology I’ve learned had a learning curve. On a carrier network I had to learn router access lists and shell scripting, and neither came easily. Moving a publishing platform onto AWS meant learning a new way to think about capacity and cost. Nobody handed me a login and expected me to be good at it the next day.

AI is the same. Giving someone access to it doesn’t mean they know how to use it. So why do so many companies roll it out as if it does?

AI replaces tasks first

AI automates tasks. Whether a job disappears depends on how much of that job those tasks were.

A job is a bundle of tasks. A support role includes answering routine questions, but also calming down an angry customer, spotting a pattern in complaints and telling engineering about it. AI is already good at the first one. It’s much weaker at the others. When most of a job is the first kind of task, that job shrinks. When it’s a mix, the job changes shape.

The research so far points the same way, and it’s worth separating what has been measured from what is being predicted.

• Measured: the Stanford Digital Economy Lab, using payroll data covering millions of US workers, found that employment of 22- to 25-year-olds in AI-exposed occupations is 19% below where it would have been had it kept pace with less-exposed peers. The decline is concentrated in occupations where AI mostly substitutes for the work. Where AI mostly complements people, employment is flat or rising, especially for experienced workers.
• Predicted: the World Economic Forum’s Future of Jobs Report projects 170 million jobs created and 92 million displaced by the end of the decade, with technology as one of several drivers. That’s a survey of over 1,000 employers about what they expect. It’s a forecast, not a measurement.

So yes, AI will replace some jobs, and the early evidence says entry-level roles feel it first. But the bigger change for most people is inside the job they already have. When producing the work gets cheap, the valuable part becomes knowing whether the work is any good.

Designers: from screens to validated experiences

I’ve written about designing every screen myself and then doing it with an AI assistant. Drawing the screen is the part that got fastest. A designer can now generate several prototypes for the same flow before lunch.

That changes where a designer’s time goes. I expect more of it to go into questions like these:

• Which of these five prototypes actually works for the people using it?
• Where in the journey do people hesitate, back out or call support?
• What do the analytics and the feedback say, and do they agree?
• When we compare two approaches, are we measuring the thing that matters?
• What did we learn, and what do we change next?

AI can help with all of that: running automated usability checks, clustering feedback, flagging drop-off points. It can’t be the user, though. Nielsen Norman Group found that AI-generated “synthetic users” are useful for generating hypotheses but tend to be overly favorable, and recommends against using them in place of research with real people.

The value moves toward knowing whether the experience works, and that takes the same skill it always did: understanding the person on the other side.

Engineers: from writing code to evaluating it

AI writes code well enough now that typing it is rarely the slow part. The slow part is everything around it. Does this design hold up at ten times the traffic? What happens when the payment provider times out? Who can call this endpoint? What does it cost to run every month? Will someone understand it in a year?

Those questions are architecture, performance, security, reliability, maintainability and operating cost. The AI will answer each one if you ask. Knowing to ask, and recognizing a weak answer, is the engineer’s job. I wrote about one side of this in recognizing AI-generated code, where a 53-line answer lost to a 6-line one.

AI is also good at the checking side. It can generate tests, run them, read the failures, suggest fixes and point out likely problems in a pull request. I use it for all of those. But there’s a question I keep coming back to:

If AI writes both the implementation and the tests, how do we know they aren’t making the same wrong assumption?

Passing tests feel like proof. They’re only proof that the code does what the tests expect. If the same misunderstanding produced both, nothing in the pipeline compares the code with what the customer asked for. The fix is a human who understands the requirement writing or reviewing the important test cases, especially the edge cases and failure paths.

Developers sense this. In Stack Overflow’s developer survey, the top frustration with AI tools was “solutions that are almost right, but not quite,” picked by 66% of respondents. In the latest edition, 48% said they trust AI when the answer is easy to validate. Trust follows the ability to check.

Product managers: from requirements to problems

AI can turn a rough idea into a tidy requirements document, a set of user stories and a clickable prototype in an afternoon. That’s useful, and it makes it easy to skip the hard part.

Building faster doesn’t mean building the right thing. If the problem is wrong, AI just gets you to the wrong product sooner, with better documentation. I’ve always believed in starting with the customer’s problem, not the architecture, and that matters more when the architecture is nearly free to produce.

So the product work that grows in value is the part AI can’t do from a prompt: talking to customers, deciding what not to build, choosing between two good ideas with one team, and connecting the work to a business result someone will pay for. How do you know this is a problem people have? What will change for them if you solve it? How will you know it worked?

Junior engineers and the learning curve

This is the part I worry about most. I learned to debug by being stuck. I read logs, broke things, fixed them, and slowly built a feel for where problems hide. If an AI writes most of a junior engineer’s code, where does that feel come from?

There’s early evidence on this. In a randomized study by Anthropic, 52 mostly junior engineers learned a new Python library, half with an AI assistant and half without. On the quiz afterward the AI group averaged 50% and the hand-coding group 67%. The biggest gap was on debugging questions. The authors call the results preliminary, and the sample was small.

The more useful finding was about how people used it. Participants who asked the AI to explain things, or who generated code and then worked to understand it, scored much higher than those who handed the whole task over. How people used the assistant mattered more than whether they had one.

That puts a responsibility on engineering leaders. If AI takes the routine tasks juniors used to learn on, we have to design the learning on purpose: reviews where they explain their code, debugging work without the assistant, smaller tasks they can own end to end, and time with someone experienced. I wrote about why I enjoy mentoring, and this is why it matters more now. If nobody grows junior engineers today, who reviews the AI’s work in ten years?

Why a broad background helps

I’ve been in this industry for 20+ years, and I’ve worked across carrier networking and VoIP, network and application security, software engineering and architecture, cloud infrastructure and DevOps, product development, and the business side of running a startup as a co-founder and CTO. I wasn’t the deepest expert in all of those. I learned how they affect each other.

That turns out to be useful with AI. When an assistant hands me a solution, I can look at it from several sides: will it scale, who can abuse it, what will it cost, who supports it, does the customer even need it? I don’t need to be a specialist to notice that a design has no rate limit or that a cost estimate ignores data transfer. I need to have seen those problems before. I went into more detail in why being a generalist is an advantage in the AI era.

That doesn’t make specialists less important. AI gets you to a decent first answer in most fields. The hard last stretch still needs people who have spent years in one area.

AI is expensive when used badly

AI doesn’t automatically make everyone more productive. In a randomized trial by METR, experienced open-source developers took about 19% longer on real tasks when AI tools were allowed. They expected a speedup beforehand and still believed they’d been faster afterward. That study used earlier tools, and newer ones are better, but the gap between feeling productive and being productive is worth taking seriously.

The costs hide in a few places: prompting in circles because the requirement was never clear, rework when the output looked right and wasn’t, extra review cycles, test suites that grow without checking anything new, and senior time spent untangling it all. Tokens are the visible part of the bill. People’s time is the larger part. I broke down the numbers in the hidden cost of AI coding.

DORA’s research on AI-assisted development puts it well: AI acts as an amplifier, magnifying an organization’s existing strengths and weaknesses. A team with clear requirements, small changes and good review gets faster. A team without them gets more of what it already had.

What engineering leaders should do

Treat AI like any other technology with a learning curve: train people, set standards, keep a human accountable, and measure results.

1. Train people as well as buying seats. A subscription is access. Show people how to write a clear task, how to check output, and when to stop prompting and think.
2. Set review and quality standards. AI-written code goes through the same review as any other code, and the person who opens the pull request has to be able to explain it.
3. Decide where a human signs off. Production changes, security-sensitive code, customer communication and anything with money or legal weight need a named person, not just a passing check.
4. Measure outcomes, not usage. Prompts sent and lines generated tell you nothing. Lead time, defects that reach customers, rework and cost per accepted change tell you something.
5. Protect sensitive information. Be clear about which tools are approved for which data, and keep customer data, credentials and confidential work out of the rest.
6. Keep investing in junior engineers. Hire them, pair them with experienced people, and give them work that builds judgment.
7. Encourage experiments, with a budget. Let teams try things, set cost limits, and share what worked and what didn’t.

None of this is new. It’s the same discipline we applied to cloud adoption, to DevOps, and to every other tool that made it easy to do a lot quickly.

Sources

Research figures were checked against these pages in October. The opinions and recommendations are mine.

• Stanford Digital Economy Lab: Canaries in the Coal Mine? Six facts about the recent employment effects of AI (revised August update)
• World Economic Forum: Future of Jobs Report (press release)
• METR: Measuring the impact of early AI tools on experienced open-source developer productivity (arXiv)
• Anthropic: How AI assistance impacts the formation of coding skills
• DORA: State of AI-assisted Software Development report
• Stack Overflow Developer Survey: AI section (previous year)
• Stack Overflow Developer Survey: AI section (latest)
• Nielsen Norman Group: Synthetic users, if, when, and how to use AI-generated research

Short answers