www.Tobold.com
Tobold@GMail.com
A blog with my thoughts regarding games I am playing and other stuff in life. Please read my Terms of Service
RSS Feed
Thursday, October 30, 2025
Work Slop
30 years ago, I wrote a Ph.D. thesis in Chemistry. It is basically a small book. As I was pretty inexperienced in writing, that book is rather badly written. But it contained the data of years of lab work, recipes on how to synthesize molecules that nobody before me had produced, and a conclusion showing how to tune the properties of an important class of catalysts. It had scientific value, at least if you believe the committee of professors who decided to award me a Ph.D. degree for it. Unfortunately, unless you have a degree in chemistry with similar specialization, the book is pretty much incomprehensible to everybody. You would need to be a specialist in the field to be able to recognize its scientific value.
While it took me months to write my thesis, and years to prepare, modern generative AI can produce a very similar looking book with an afternoon's amount of work of somebody prompting it. It would be equally incomprehensible to most people, but look nicer, and appear to be much better written. Only a detailed examination by a specialist would reveal that the document holds no scientific value at all. By definition, something written by generative AI is either copied from somewhere, or hallucinated. It is work slop, like Deloitte's AI-written report for the Australian government. Looks good, until you verify the data and the citations, at which point the hallucinations become obvious.
Most of my life's work resulted in written documents: My thesis, several scientific publications, countless technical reports, and 32 patents. Arguably I would have loved an AI to help me write the fluffier parts, but the value of those documents is all in my data and my work they report. You can get generative AI to write you a patent for a perpetuum mobile, but that doesn't mean that you'll have a valid invention on your hands. Scientific documents, legal documents, engineering documents, everything where the value lies in facts being reported correctly, can be faked by generative AI. But unless you only let the generative AI write the prose and let the expert write the data and conclusion from those data, the AI document is worthless.
I am pretty sure that all my patents have been verified and checked by an expert, because as an inventor you get the comments from the patent office examiner back. Scientific publications are peer reviewed, and a committee verifies any thesis before awarding a degree. With my technical reports I wrote for internal use at the company I worked for, I am not so sure. Most people probably only read the cover page, and detailed scrutiny by another expert is unlikely. If Deloitte dares to deliver an AI-written report to a government that paid them, I can easily imagine somebody in a company writing his reports with AI without making sure that all the facts are correct. Work slop looks nice, it just doesn't hold any actual value. Just like AI generated images are getting harder and harder to spot directly, AI generated written documents aren't immediately obvious. If somebody got through college with AI-generated essays, why wouldn't he deliver AI generated work slop in a company, when his supervisor asks him to quickly produce some report or other document? Management might actually encourage the use of AI, not understanding its limitations. How likely is the employee to apply critical thinking to the AI generated sales report? Would he spend hours verifying, when the computer produced the documents in minutes? Work slop has the potential to do serious harm to companies that end up following hallucinated conclusions, because nobody bothered to verify how they got there.
- posted by Tobold @ 2:12 PM Permanent Link
Comments:
<< Home
Newer› ‹Older
Tobold, are these true statements? AI can currently only produce answers to questions that have already been answered. Thus, the current state of AI is an encyclopedia that looks up answers for the prompter.
If true, one can't help but believe unless it improves significantly by 2030, there is an AI stock market crash coming.
If true, one can't help but believe unless it improves significantly by 2030, there is an AI stock market crash coming.
# posted by 
: 30/10/25 18:12
: 30/10/25 18:12
I would think that an encyclopedia would actually be more likely to give you a correct answer. Generative AI works by putting words together in previously seen patterns. Thus you *might* get a previously seen answer to your question, with the probability being high if that question was answered frequently in the training data. But the AI will never answer “Oh, I don’t know that”, and so if your question is obscure, you might get an answer that is just made up.
By the way, Nvidia today passes a market cap of $5 trillion, which incidentally is also the estimated GDP of Germany this year. You decide whether Nvidia is worth as much as the whole economic output of a large industrialized country.
The problem with AI is that it neither knows nor cares whether any of the ouput it produces is accurate, while at the same time it is very good at presenting the output in a way that it will appear to be accurate at a first glance. When it comes to output that requieres an expert to really fact check, a first glance is (essentially) all that most peaple are capable of. The inevitable result of this is going to be a lot of factually innacurate passages of prose being mistaken for the truth.
The current generative AI is not an encyclopedia that copy pastes the search result back to you. That would be better.
It is a text (or rather token) predictor that takes your input and looks up what other tokens have been seen in the training data in that context.
Each token has a probability to appear and the AI is picking the things that have a high probability (not sure if the highest, but likely not) and adds it to the answer.
That's where the hallucinations come from. It has seen specific tokens and chains those together. You are not asking the AI to think, you are asking it to generate a token chain. It cannot reply with "I don't know" - unless those were in the training data with a high frequency related to your input. Even then it is just generating that text and not thinking.
If you remember the seahorse incident, then that is basically what happens with tokens. A technical post about it: https://vgel.me/posts/seahorse/
Or just check the Twitter link at the top for some giggles.
It is a text (or rather token) predictor that takes your input and looks up what other tokens have been seen in the training data in that context.
Each token has a probability to appear and the AI is picking the things that have a high probability (not sure if the highest, but likely not) and adds it to the answer.
That's where the hallucinations come from. It has seen specific tokens and chains those together. You are not asking the AI to think, you are asking it to generate a token chain. It cannot reply with "I don't know" - unless those were in the training data with a high frequency related to your input. Even then it is just generating that text and not thinking.
If you remember the seahorse incident, then that is basically what happens with tokens. A technical post about it: https://vgel.me/posts/seahorse/
Or just check the Twitter link at the top for some giggles.
Cmon Tobold, I know you’re sceptical of AI and rightly so, but comparing market cap with GDP is a nonsense comparison. The VALUE of Germany is something very different from its GDP. So either compare market cap with value of all assets in the country or yearly turnover with GDP!
# posted by : 1/11/25 03:08
: 1/11/25 03:08
Why not? I could compare my annual salary with the price of a Lamborghini, although one is an income and the other a value. I didn't say that Nvidia was worth as much as Germany, I said it was worth as much as Germany's economic annual output.
Post a Comment
<< Home
