Trusting Gen AI with career decisions has pros and cons. It excels at personalized insights, unbiased feedback (if trained well), and real-time updates. For math teaching, its adaptability to data and trends can make it highly effective. However, it may lack emotional sensitivity, rely on biased training data, or fail to consider nuanced personal goals.

Over-reliance could also undermine critical thinking, and impersonal guidance can't replace human mentorship. GenAI works best as a tool to supplement decisions—not replace human judgment or introspection.

Should there always be a human in the loop for such critical life choices?

Advantages of Trusting GenAI with Career Decisions

Personalized Insights: Tailored advice based on performance metrics and goals.

Scalability: Accessible to a wide audience, bridging gaps where human mentors aren't available.

Unbiased Feedback: Neutral guidance if trained on diverse, unbiased data.

Real-Time Updates: Keeps pace with evolving industry trends.

Continuous Improvement: Learns and adapts through user feedback.

Data-Driven Decision-Making: Offers recommendations grounded in large-scale data analysis.

The above answer is AI answering an AI question! Easily identified now, but AI is rapidly advancing where one might not know. My impression at this point in time is AI uses stilted language, often too complex if the person conducted the writing without AI, and still misses the point. Of course, a skilled AI operator knows answers rely on the input, so human clarification of what it wants from AI is paramount. I have a friend who teaches at another college, and 100% of discussions and responses are AI. Easily identified because complete novices could never have such thorough answers and read a half dozen textbooks in a day. Their responses all have the same identifiers and word structure. [Students who use the same AI platforms have remarkably similar answers and citations!]

Great question Edward Willett At the heart of this dilemma you have the same bothersome problems.

So let's look at some in no particular order

• Does a coach or mentor need a set of morals? It would seem that succeeding in life or business etc requires Emotional Intelligence. Could a soulless AI foster altruism and caring in its human client? But maybe in academia having a conscience is not really necessary;_) Would the AI coach need a human chaplain or the three laws of robotics?
• Then there is a crucial aspect Can AI avoid GIGO garbage in garbage out? Looking at the information online of the last 12 years of American politics I daresay AI's advice might have at the very least been capricious. Science is worse, over the last thirty years' research has been left undigested. What I mean to say is: where are the paradigm shifts that should have occurred or reinterpretations of theories from the twentieth century? For example, there is not a single planetary atmosphere held in place by gravity. This idea dates from Newton's time.
• In the same vein, AI can augment or amplify human intents. This is for sale and isn't money the root of all evil? Plus the even bigger picture. AI and the networks of cyberpeople (connected humans and chatbots) are akin to demons and angels pulsating posts, articles, and messages similar to the minds of termites. David C. Coker is right whither they will evolve?
• I type faster because I played Marvis Beacon teaches typing. Therefore undeniable AI has its place in education but how and where is still in the air like allowing calculators in math class?
• Of course, AI can write books, symphonies, or catch pop tunes. Wildly amazing but Eldritch just the same.

Just some thoughts while shaving

Yours Prof YUKNA

PS. I wrote this with Grammarly and while the spellcheck function was well-appreciated, I constantly had to wrestle with it to keep my voice or nuances of my ideas intact.

Hello

See my review of Mollick's "Co-Intelligence: Living and Working with AI" as he provides some helpful discussion and examples related to utilizing AI as an assistant:

https://www.amazon.com/review/R30AYUAIY3A47Q/ref=cm_cr_srp_d_rdp_perm?ie=UTF8

All the best,

Fred

Thank you. I will take a look. E

Researchers find "certain principles – openness, education, legality, and mutual respect – were violated in scenarios with "AI-based research mentors."...

AI-based research mentors: Plausible scenarios and ethical issues

"Guided by Brey’s Anticipatory Technology Ethics, we examined AI-based research mentors (AIRMs) through technology foresight as well as identification and evaluation of ethical issues. Scenario planning was employed to inform foresight, yielding four plausible future scenarios: 1) AIRMs are used solely for guidance, 2) AIRMs are used for guidance and monitoring, 3) AIRMs are banned, and 4) AIRMs are used solely for monitoring. Resnik’s twelve principles informed the identification of ethical issues within these scenarios. Our analysis revealed that certain principles – openness, education, legality, and mutual respect – were violated in all scenarios. Others were contravened to varying degrees across the scenarios; for example, freedom was only violated in scenarios where AIRMs were used for monitoring. Furthermore, the guidance scenario showed that AIRM’s responses could be manipulated to justify poor practice (“AIRMing”). In our evaluation, we weighed ethical issues against the benefits and found that the guidance-only scenario was the least problematic. While this scenario has benefits, such as providing expert guidance on research, ethical issues arise with regard to honesty, openness, credit, education, legality, and mutual respect. Therefore, policy must be developed to ensure that AIRMs are used solely for guidance while mitigating these issues."

https://www.tandfonline.com/doi/full/10.1080/08989621.2025.2551890