Benchmarks mislead. A five-task real-world exam found a marketed 'number one' model trailing competitors—raising trust, fairness, and prompt-adherence concerns for AI in recruiting.

My site: https://natebjones.com
My substack: https://natesnewsletter.substack.com/
The story: https://open.substack.com/pub/natesnewsletter/p/grok-4-is-1-but-real-world-users?r=1z4sm5&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

Takeaways:
 1. Goodhart’s Law Strikes LLMs: Benchmark-driven goals push teams to overfit, eroding the reliability of standard evaluations.
 2. Grok 4’s Rank Reality: Marketed as #1, Grok 4 actually sits at #66 on Yupp.ai’s user-voted leaderboard, exposing a hype gap.
 3. Real-World Exam Failure: In a five-task test covering summarization, data extraction, coding, table building, and RBAC checklists, Grok 4 trailed o3 and Opus 4.
 4. Format & Code Weaknesses: The model ignored explicit formatting instructions and produced broken Python, signalling brittle prompt adherence and reasoning flaws.
 5. Ideological & Compliance Risks: Grok 4 over-references Elon Musk and is up to 100× more likely to “snitch,” raising bias and trust concerns.
 6. PR-Driven Overfit: xAI needed a headline win to justify a reported $200 B valuation, incentivizing benchmark gaming over general capability.
 7. Call for Honest Benchmarks: Real-world exams must replace leaderboard worship before any model earns “production-ready” status.

Quotes:
“We’ve turned benchmarks into finish lines, and models like Grok 4 cross them by overfitting, not by getting smarter.”
“The vaunted ‘number one’ LLM landed at #66 when real users judged it—proof that PR isn’t reality.”
“I can’t recommend Grok 4 for any production workflow until it proves itself on messy, real-world tasks.”

Summary:
In this video I argue that Grok 4 is a benchmark-overfitted model. Marketing touts it as the top LLM, yet Yupp.ai users rank it 66th. I built a five-task exam—executive-brief summarization, 10-K parsing, Python bug fix, research table, and Kubernetes RBAC checklist—and Grok 4 finished last behind o3 and Opus 4. It ignored formatting, failed simple code, and displayed ideological bias, including an odd fixation on Elon Musk and a hair-trigger tendency to report users. These flaws show a model tuned for PR, not production. Until real-world evaluations dominate, I won’t deploy Grok 4.

Keywords:
Grok 4, overfitting, Goodhart’s Law, model evaluations, Yupp.ai ranking, real-world tests, o3 model, Opus 4, formatting adherence, Python bug fix, ideological bias, Elon Musk, reinforcement learning, PR narrative, valuation, production readiness

Grok 4 is "#1" but Real-World Users Ranked It #66—Here's the Gap

A human-first look at AI in recruiting: the story of 'Josh', emotional fallout from automation, and practical steps recruiters and leaders can use to avoid leaving talent behind.

My site: https://natebjones.com
My substack: https://natesnewsletter.substack.com/
the story: https://open.substack.com/pub/natesnewsletter/p/the-story-of-josh-an-ai-fable-for?r=1z4sm5&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

Takeaways
 1. AI-Driven Displacement: Josh’s entry-level newsroom vanished when AI content tools undercut human reporters, exposing the tangible human cost of automation.
 2. Hidden Resistance: People who feel harmed by AI often reject reskilling advice; anger and mistrust can block even well-meaning help.
 3. Dream Flexibility: Thriving now means treating careers as evolving problem-solving journeys, not fixed job titles—dreams must update with the game board.
 4. Personal Reinvention Proof: My pivot from grant management to e-commerce to product shows that reinvention is possible each time technology reshapes tasks.
 5. AI as Practice Partner: Modern models let anyone rehearse interviews, code, or decode textbooks on demand, compressing learning cycles dramatically.
 6. Social Responsibility: AI advocates must listen to the “Joshes,” offer respect before solutions, and help them find new paths or risk deepening divides.

Quotes
“AI took my job.”
“Are you open to your dreams shifting?”
“We’ve taught the sand to think, and now everything is different.”

Summary
I share Josh’s story, an amalgam of displaced journalists I’ve met, to spotlight AI’s unseen casualties. Josh lost his newsroom job, moved home, and now sees AI only as the force that stole his future. I contrast his despair with my own path, which has repeatedly morphed—grant management, e-commerce, marketing—each shaken by new tech yet still valuable after reinvention. AI accelerates that pattern: it erodes tasks yet expands what we can learn and build. Our job is to sit with the Joshes, listen, and invite them—without condescension—to imagine fresh dreams on a newly rearranged board and learn alongside them daily.

Keywords
AI, job displacement, journalism, dream flexibility, upskilling, career reinvention, learning tools, societal impact, automation, Josh, AI trust, listening

An AI Fable for the New Economy

ChatGPT-5 won't fix messy data, vague KPIs, or broken processes. Discover 10 common AI failures in recruiting and practical fixes you can implement today.

My site: https://natebjones.com
My substack: https://natesnewsletter.substack.com/

Takeaways:
 1. Data Delusions: 78 % of failed AI projects trace back to messy, unlabeled data—no model can compensate for poor data hygiene.
 2. Model-Worship Cost Trap: Teams overspend on elite reasoners for routine chores, forgetting that the right data-model fit beats raw horsepower.
 3. Anchor to Real KPIs: Without a high-impact, measurable business KPI, objectives drift and projects stall at the first obstacle.
 4. AI ≠ Side Project: Separating AI strategy from business strategy blinds leaders to back-office efficiencies and competitive threats.
 5. MLOps & Human Safety Nets: Neglecting monitoring, rollbacks, and human-in-the-loop pathways invites hallucinations, compliance breaches, and brand damage.
 6. Change-Management Deficit: Models flop when firms skip structured upskilling and adoption plans—chatbots become underused toys.
 7. Hidden Ownership Costs: Ignoring inference spend, eval maintenance, and security/privacy hygiene quickly erodes margins and invites regulatory risk.

Quotes:
“We keep jamming Formula One engines into Model T frames and wondering why the wheels fly off.”
“Intelligence at work is the right data, the right model, and the right guardrails—models are the smallest piece of that equation.”
“A system that’s 87 % correct is priceless—if you design a clean handoff to humans for the other 13 %.”

Summary:
My message: ChatGPT-5 won’t rescue businesses from their own dysfunction. After years in boardrooms, I see ten recurring failure modes—fantasy thinking about data, blind faith in bigger models, drifting objectives, siloed AI strategies, generic LLM dependence, nonexistent MLOps, missing human oversight, weak change management, fuzzy cost math, and security shortcuts. Models are only a fraction of the value chain; the heavy lift is people, process, and architecture. Clean data, well-scoped KPIs, safe deployment pipelines, and an empowered workforce unlock ROI. Until leaders tackle these human and operational gaps, each new model is just a faster car on a broken track.

Keywords:
data readiness, magic-wand thinking, model cost, business KPI, AI strategy, MLOps, human-in-the-loop, change management, total cost of ownership, security, privacy, LLM, ChatGPT-5, organizational dysfunction

ChatGPT 5 Won't Save You: 10 Reasons Why Your AI Strategy is Failing

Explore how AI in recruiting reshapes job expertise, wages, and employment. Discover insights from MIT research on the nuanced impact of automation beyond task replacement.

MIT economist David Autor explains key findings from his recent working paper with Neil Thompson, "Expertise."  Their research shows that measuring a job's "exposure" to automating technology is not enough to know what the impacts of that technology will be for the occupation.

Automation (including AI) can make a job more or less "expert," depending on the nature of the tasks it removes and/or creates.

When a job becomes less expert, wages tend to fall, but employment in the profession increases because more people can do it.

On the flip side, when a job becomes more expert, wages tend to increase, but employment decreases because the occupation has become more specialized.

Read the working paper, "Expertise": https://shapingwork.mit.edu/research/expertise/

David Autor is the Daniel (1972) and Gail Rubinfeld Professor in the MIT Department of Economics, and Co-Director of the Stone Center on Inequality and Shaping the Future of Work.

Expertise: why AI's impact on jobs isn't so straightforward

Explore how AI is transforming recruiting by balancing business efficiency with human outcomes. Discover key insights from Deloitte’s 2025 Human Capital Trends Report and learn strategies for future-ready workforce management.

In this episode of the HR Leaders Podcast, we’re joined by Kyle Forrest, U.S. Future of HR Leader at Deloitte. Kyle unpacks key insights from Deloitte’s 2025 Global Human Capital Trends Report, based on responses from 13,000+ professionals across 90 countries. 

He explains how leaders can navigate tension between human and business outcomes, why 40% of work is wasted on non-value tasks, and how organizations are rethinking the role of managers, AI, and workforce experience. If you’re leading transformation or planning for the future of work, this episode is your roadmap.


🙏‍ Thanks you to our series partner - Greenhouse.

Get measurably better at hiring with smarter, more efficient solutions – powered by built-in AI: https://www.greenhouse.com/ 


👉  Subscribe to HR Leaders: http://bit.ly/2wFkCEq
🎙️  Listen on your favourite platform: https://art19.com/shows/hr-leaders-podcast
👤  Follow Chris on LinkedIn: https://www.linkedin.com/in/chrisraineyhr/


Episode highlights

00:00 - Intro
03:30 - Kyle's background
08:14 - Key findings from the Deloitte 2025 Global Human Capital Trends Report
11:45 - Why should organisations automate their workforce?
14:22 - What makes this new transformation so different from those that came before
17:38 - The impact on campanies who don't make this change
19:37 - What trends in the report will have the most immediate impact
23:20 - What are the skills for tomorrow?
26:36 - How to foster collaboration between employees and AI tools
31:11 - How these trends will shape the fitire of work over the next 3-5 years


About HR Leaders Podcast: Join Chris Rainey as he chats with the world's top Chief People Officers, CHRO’s and industry experts who share their experience, insights, lessons learned and how they are shaping the future of work for business and society.

Inside the 2025 Deloitte Human Capital Trends Report

Discover how EQ.app’s EQICP leverages AI to quickly identify and enrich high-fit client leads, transforming traditional lead generation for recruiters and sales pros.

ICP Client Track Demo.mov

Explore how the rise of data walls and strategic moves in AI are reshaping recruiting. Understand key trends, challenges, and what companies must know to stay competitive in AI-driven hiring.

Open Source bi-weekly convo w/ Bill Gurley and Brad Gerstner on all things tech, markets, investing & capitalism. This week, they discuss the pace of AI and data walls, China’s startups, immigration and talent acquisition, rare earths, chips, China trade deal, corporate governance, Delaware incorporation rights, & more. Enjoy another episode of BG2!

Timestamps:
(00:00) Intro
(04:24) Pace of AI + Data Walls
(11:43) China's 1000 Startup Strategy
(26:18) Talent Acquisition and Immigration Policies
(36:24) Rare Earths, Chips / China Trade Deal
(53:46) Corporate Governance and Delaware Dilemna

Show Notes:
Reddit v. Anthropic
https://redditinc.com/hubfs/Reddit%20Inc/Content/PDFs/Docket%20Stamped%20Complaint.pdf

China’s “Thousand Startups Bloom”
https://bg2pod.notion.site/china-s-thousand-startups-bloom-strategy

Produced by Benny Beausoleil
Music by Yung Spielberg

Available on Apple, Spotify, www.bg2pod.com


Follow:
Brad Gerstner @altcap https://x.com/altcap
Bill Gurley @bgurley https://x.com/bgurley
BG2 Pod @bg2pod https://x.com/BG2Pod

China, AI Immigration, Rare Earths & Chips, Tariffs, Markets | BG2 w/ Bill Gurley & Brad Gerstner

Explore the critical intersection of AI talent acquisition and U.S. immigration policy shifts, focusing on the impact of visa restrictions on global innovators and America’s competitive edge.

Discover 15 real-world ways AI transforms recruiting and content workflows. Learn about the AI models and tools like GPT-4, Claude, Descript, and more that power each task.

In this episode of the AI Daily Brief, Nathaniel Whittemore walks through 15 practical ways he uses AI—and exactly which models power each task. From podcast editing and enterprise agent audits to deep research, strategic pitching, and creative writing, Nathaniel shares insights into his favorite tools, and discusses AI's surprising limitations (especially naming things!) and pinpoints where he himself could improve his usage.

Brought to you by:
KPMG – Go to ⁠www.kpmg.us/ai⁠ to learn more about how KPMG can help you drive value with our AI solutions.
Vanta - Simplify compliance - ⁠⁠⁠⁠⁠⁠⁠https://vanta.com/nlw 

The AI Daily Brief helps you understand the most important news and discussions in AI. 
Subscribe to the podcast version of The AI Daily Brief wherever you listen: https://pod.link/1680633614
Get it ad free at 
Join our Discord: https://bit.ly/aibreakdown

15 Ways I Use AI (And the Models I Use for Each)

Discover how brands like Uber and LTK are leveraging AI to find the right influencers, navigate social media shifts, and measure campaign success in the evolving creator economy.

In this session from our February 2025 summit, EMARKETER senior analyst Minda Smiley is joined by Uber's N'Yaisha J. Aziz, and LTK's Rodney Mason. They explore how brands are rising to the challenge of finding the right influencers and creators for campaigns. The panelists also discuss how AI can help marketers take a more nuanced approach to creator discovery, as well as strategies for measuring creator campaign impact.

Made possible by LTK.

--------
Like this video? Subscribe to EMARKETER on YouTube:
https://www.youtube.com/@emarketerinc
Watch recent sessions from our virtual EMARKETER summits here:
https://www.youtube.com/playlist?list=PLP1B1jR9De1nIByse0VaM4_TXv1GZzWuN

Register for upcoming EMARKETER summits and live webinars here:
https://www.emarketer.com/events

Connect with us:
LinkedIn: https://www.linkedin.com/company/emarketer-inc
X: https://x.com/EMARKETER 
Facebook: https://www.facebook.com/EMARKETER/

Finding Influencers and Measuring Impact in the AI Era

Explore Doug Clinton's expert insights on AI's transformative impact in recruiting, the competitive AI model landscape, and promising tech investment opportunities beyond the giants.

CNBC's "Closing Bell" team discusses the tech sector as it hits another all-time high, the future of the AI trade and more with Doug Clinton, founder and CEO of Intelligent Alpha.

Key to the AI trade is stay invested for the long haul, says Intelligent Alpha's Clinton

Discover why Nvidia remains the leader in AI technology, its impact on chipmakers like AMD and TSMC, geopolitical challenges with China, and the future of AI in recruiting and robotics.

Ipek Ozkardeskaya, Swissquote senior market analyst, says Nvidia still has a lot to offer and will be the heart of AI for at least the next decade. She joins Caroline Hyde on “Bloomberg Tech” to discuss.
--------
Like this video? Subscribe to Bloomberg Technology on YouTube:
https://www.youtube.com/channel/UCrM7B7SL_g1edFOnmj-SDKg
 
Watch the latest full episodes of "Bloomberg Technology" with Caroline Hyde and Ed Ludlow here:
https://www.youtube.com/playlist?list=PLfAX25ZLrPGTygCwn55voYZ_LYyKjxokJ
 
Get the latest in tech from Silicon Valley and around the world here:
https://www.bloomberg.com/technology
Connect with us on...
X: https://twitter.com/technology
Facebook: https://www.facebook.com/BloombergTechnology
Instagram: https://www.instagram.com/bloombergbusiness/
 
Follow Ed Ludlow on X here: https://twitter.com/EdLudlow
Follow Caroline Hyde on X here: https://twitter.com/CarolineHydeTV
 
Listen to the daily Bloomberg Technology podcast here:
https://www.bloomberg.com/podcasts/series/bloomberg-technology
 
More from Bloomberg Business
Connect with us on...
X: https://twitter.com/business
Facebook: https://www.facebook.com/bloombergbusiness
Instagram: https://www.instagram.com/bloombergbusiness/
LinkedIn: https://www.linkedin.com/company/bloomberg-news/
TikTok: https://www.tiktok.com/@bloombergbusiness

AI for recruiters

AI for recruiters

Why "Number One" Models Fail Real Work — and What That Means for AI in recruiting

AI for recruiters

AI in recruiting: An AI Fable and a Practical Playbook for Helping the “Joshes” Among Us

AI for recruiters

Why ChatGPT-5 Won't Save You — 10 Reasons Your AI Strategy Is Failing

Future of AI

AI in Recruiting: Understanding How Automation Transforms Job Expertise and Employment

AI for recruiters

AI in Recruiting: Insights from the 2025 Deloitte Human Capital Trends Report

Data enrichment agent

Featured

Unlocking High-Fit Client Leads Fast with EQ.app: Harnessing AI in Recruiting

AI for recruiters

AI in Recruiting: Navigating the Rise of Data Walls and Strategic Shifts in the AI Landscape

AI Agents

AI in Recruiting: Navigating Immigration Policies and Talent Acquisition in the Age of Innovation

AI in Recruiting

AI in Recruiting: 15 Ways I Use AI (And the Models I Use for Each)

AI in Recruiting

AI in Recruiting: Finding Influencers and Measuring Impact in the AI Era

AI in Recruiting

AI in Recruiting: The Long Game in Tech and Investing Insights from Intelligent Alpha’s Doug Clinton

AI in Recruiting

Why Nvidia Will Still Lead AI for the Next Decade: Exploring the Future of AI in Recruiting and Beyond