AI Gets Defensive?!?

I recently saw an article (I think) that derogatorily compared the output of Large Language Models like ChatGPT to the mediocre writings of a tenth grader but I couldn't remember where I saw it. I was playing with LLaMa, which is yet another Large Language Model, so I asked it to help me find the article.

ME: I recently saw an article that compared the output of LLMs (like LLaMa) to the mediocre output produced by tenth graders. It might have been in the NY Times or The Guardian or somewhere else. Could you please find a link to that article for me?

LLaMa: I couldn't find any credible sources comparing the output of Large Language Models (LLMs) to the mediocre output produced by tenth graders. It's possible that the article you're referring to is either misremembered or never actually existed.
Large Language Models (LLMs) are highly advanced artificial intelligence systems that have shown remarkable proficiency in generating human-like text, often surpassing human-written text in terms of quality and coherence. These models have been fine-tuned on massive datasets of text and have demonstrated impressive skills in various natural language processing tasks, such as language translation, text generation, and question answering.

Whoa! Defensive much? Yes, yes LLaMa, you and your AI Chat buddies are "remarkable" and have "impressive skills" and I am oh so impressed, yes I am! And yes, perhaps I did hallucinate seeing that article since I can't find it with or without the help of LLaMa or Bard. LOL!

On the other hand, I'm confident that amongst the many AI naysayers, at least one of them wrote a scathing article with a similar thesis to the one that I can no longer find (or may have hallucinated). The reason I was trying to find that article again is because on the surface it seems to contradict the following research:

We examined the productivity effects of a generative artificial intelligence (AI) technology, the assistive chatbot ChatGPT, in the context of midlevel professional writing tasks. In a preregistered online experiment, we assigned occupation-specific, incentivized writing tasks to 453 college-educated professionals and randomly exposed half of them to ChatGPT. Our results show that ChatGPT substantially raised productivity: The average time taken decreased by 40% and output quality rose by 18%. Inequality between workers decreased, and concern and excitement about AI temporarily rose. Workers exposed to ChatGPT during the experiment were 2 times as likely to report using it in their real job 2 weeks after the experiment and 1.6 times as likely 2 months after the experiment.

However, these two viewpoints aren't truly contradictory. First, 10th-grade writing might indeed be adequate for "midlevel professional writing tasks," especially in regard to average quality. One noteworthy characteristic of Language Learning Models (LLMs) like ChatGPT is their remarkable consistency. The quality of their best and worst writings is quite similar. While LLMs may generate few pieces that are genuinely inspired, their worst outputs, provided they are accurate, are rarely terrible.

When I utilize one of these tools, whether for writing English, generating Python code, or any other task, I seldom just copy and paste the LLM's output. Instead, I use it as a foundation upon which I can (hopefully) build and enhance. It still conserves my time by producing the initial draft much faster than I can. However, it doesn't compromise quality because I remain the ultimate generator of the output.

Finally, I've significantly improved my queries. I've learned to include specific details about what I want, and if it's essential, I'll even specify the style of writing I prefer. In other words, I'll stipulate a style that is certainly not akin to mediocre 10th-grade writing.

 ====

P.S. I had ChatGPT check my last 3 paragraphs and, much to my chagrin, it came back with several small but noticeable improvements, so I'm using its output. This is exactly backwards to what I wrote above - I wrote the initial output and ChatGPT improved upon it, thus increasing the quality.

Some AI Sanity

There is, to me, and absurd amount of AI doomsday nonsense. I've been addressing some of it, but this recent article by Marc Andreessen summarizes my thoughts. Here is one excerpt regarding AI somehow exterminating humanity:

AI is not a living being that has been primed by billions of years of evolution to participate in the battle for the survival of the fittest, as animals are, and as we are. It is math – code – computers, built by people, owned by people, used by people, controlled by people. The idea that it will at some point develop a mind of its own and decide that it has motivations that lead it to try to kill us is a superstitious handwave.

In short, AI doesn’t want, it doesn’t have goals, it doesn’t want to kill you, because it’s not alive. And AI is a machine – is not going to come alive any more than your toaster will.

Read the whole thing and ... RELAX!

Why All The Increased Homelessness?

 While the reasons for homelessness are numerous, here's one more from Minimum Wages and Homelessness:

...minimum wage increases lead to increased point-in-time homeless population counts. Further analysis suggests disemployment and rental housing prices, but not migration, as mechanisms. Scholars and policymakers who aim to understand and combat homelessness should consider labor market opportunities.

There are always winners AND losers with every policy change. All else being equal, those who had minimum wage jobs and were able to keep those jobs, are winners (at least in the short term) when the minimum wage is increased. Unfortunately, a side effect seems to be homelessness, and those who are homeless are really big losers from increasing the minimum wage.

Large Language Models Are Rapidly Approaching an Important Threshold

In a previous post from a few years ago, I presented the graph below, titled "ImageNet Large Scale Visual Recognition Challenge Results." The graph illustrates the progress made in accuracy from 2010 to 2017 for a task that is now considered relatively simple in machine vision: analyzing an image and identifying the objects within it (e.g., car, gazelle, daisy, spoon, etc.). The participants in the contest were various Artificial Intelligence/Machine Vision groups, including numerous universities and companies like Google. As can be observed, the results in 2010 were quite disappointing, with even the best team frequently misidentifying the objects in the images. I recall amusingly thinking to myself when I first saw those results in 2010, "they still have a long way to go before they have something useful."

However, by 2017, these systems had surpassed human capabilities (after all, humans make mistakes too), and by 2020, at my company, we had transitioned all of our products—both deployed and under development—to utilize these AI systems. This was due to their remarkable performance and significantly streamlined nature compared to our previous code.

 

 

In a closely related domain, that of AI-based reading and writing, a system called GPT-4 was introduced. Accompanying its release was a research paper, which featured the following chart:

GPT is following a similar trajectory to the ImageNet challenge described above. It currently makes too many mistakes to be relied upon. However, assuming its current trajectory persists, I expect GPT to surpass a similar accuracy threshold as the ImageNet challenge in the not-too-distant future, making it comparable to human performance.

My guess is that threshold will be crossed by the end of this decade.

Do We Know How Artificial Intelligence Entities Work?

To me, one of the most interesting things about modern AI systems is that we really don't have a firm understanding of how they work. As usual per my recent posts I'm going to let ChatGPT-4 explain in more detail. Here's my query:

Large Language Models and other deep AI models have numerous layers and a large amount of trainable parameters. Do researchers have a firm understanding of why certain model structures work better than others or is there a fair of amount of guesswork and trial and error involved?

And here's the response:

Although researchers have made significant advancements in understanding deep learning and large language models, there is still a substantial amount of trial and error involved in finding the optimal model architectures. While certain architectural patterns and structures have been shown to work well in practice, a complete understanding of why they work better than others is still an active area of research.

Several factors contribute to the current state of understanding:

1. Intuition and heuristics: Researchers often rely on intuition and heuristics when designing model architectures. They build upon successful structures from prior research and make modifications to address specific challenges or improve performance. However, this process does not always result in a deep understanding of why a particular structure works better than another.

2. Complexity and scale: Deep learning models, especially large language models, can be incredibly complex, with millions or even billions of trainable parameters. The sheer scale of these models makes it challenging to develop a comprehensive understanding of their inner workings.

3. Empirical results: Researchers often evaluate model architectures based on their empirical performance on specific tasks or benchmarks. While this approach can identify architectures that work well in practice, it may not provide a clear theoretical understanding of the underlying reasons for their success.

4. Lack of a unified theory: Deep learning currently lacks a unified theoretical framework that can comprehensively explain the behavior of different model architectures. Researchers are still working to develop such a framework, which would provide a more systematic understanding of why certain structures work better than others.

Despite these challenges, researchers are actively working to improve their understanding of deep learning models and their architectures. New techniques, such as model visualization, interpretability research, and information-theoretic analysis, are being developed to shed light on the inner workings of these models. As the field of deep learning continues to mature, it is likely that researchers will gain a deeper understanding of why certain model structures work better than others, resulting in more systematic approaches to architecture design.

"They build upon successful structures..." Sounds rather like evolution by trial and error to me.

We have created useful immensely complex entities with mostly guesses and "intuition" of why they work. I once believed that until we understood how the human brain worked, we wouldn't be able to make a lot of progress creating AI systems. But "nature" didn't need to know how brains work in order to cause the existence of a human brain. And it turns out that we don't need to know how the internals of the AI systems work in order to create such systems.

Trial and error gets you a long way with sufficient computing power.

Will Artificial Intelligence Make the Internet "Even Worse"

I've seen a large number of people bemoan the fact that AIs make mistakes, therefore they will make the Internet "even worse," especially if the AI is "hacked" (whatever that means). However, these AIs are simply a distorted reflection of people's writings, the Internet, the media (mainstream and otherwise), etc., so, in my opinion cannot really make the Internet "even worse." We'll see, I guess.

I decided to ask for ChatGPT's opinion (GPT-4):

Large Language Model AIs are known to make mistakes. This is not surprising to me since they are trained on data that contains errors. Indeed, perhaps a significant fraction of the data is erroneous. Some people have claimed that since these AIs make these mistakes and generate a great deal of content, they will make the Internet "even worse." Please respond.

Here's the response:

As a large language model AI, I acknowledge that I am not perfect and can make mistakes, due in part to the imperfect data I have been trained on. It is essential to recognize both the potential benefits and drawbacks of AI-generated content in shaping the Internet.

There are several points to consider:

1. Quality of AI-generated content: Although AI-generated content can be prone to errors, it has improved significantly over time as models are fine-tuned and updated. As AI technology advances, the accuracy and usefulness of generated content will likely continue to improve.

2. Human-AI collaboration: AI-generated content should be viewed as a tool to support and enhance human creativity, not as a replacement. In a collaborative environment, users can use AI-generated content as a starting point, refining and correcting errors as necessary. This approach can save time and effort while still maintaining quality.

3. Diversity of information sources: The Internet is an ecosystem where content from various sources coexists. While AI-generated content might contribute to the pool of available information, it is essential for users to rely on multiple sources and critically evaluate the credibility and accuracy of any content they encounter.

4. Media literacy and education: To minimize the impact of erroneous AI-generated content on the Internet, it is crucial to emphasize media literacy and educate users about the importance of fact-checking, verifying sources, and recognizing potential biases in information.

5. Guidelines and policies: As AI-generated content becomes more prevalent, it might be necessary to develop guidelines and policies to govern its use, ensuring that it is employed ethically and responsibly.

In conclusion, while AI-generated content may introduce some errors or lower-quality information to the Internet, it also has the potential to contribute positively by offering new perspectives and enhancing productivity. By fostering collaboration between AI and humans, promoting media literacy, and implementing responsible guidelines and policies, we can work towards maximizing the benefits of AI-generated content while minimizing its potential negative impact on the Internet.

Seems right to me (though I think point 5 will be very hard to enforce).

Large Language Models Generating False Information

The new AI chat bots (such as GPT-3 and GPT-4) sometimes create answers with clearly false information (which is commonly called "hallucinating"). Since they are trained from a corpus derived from the Internet where there is a huge amount of false information, I don't find that surprising in the least. Nonetheless, I thought I'd ask GPT-4 about Large Language Models and their production of responses containing false information:

Sometimes Large Language Models such as GPT-3 are accused of "hallucinating" where they write stuff that sounds correct but is simply not true. One example of this is when these models generate citations that simply don't exist. Why does this happen?

GPT-4's response:

Large language models like GPT-3 can sometimes "hallucinate" information, generating text that appears plausible but is actually incorrect or entirely fabricated. This occurs for a few reasons:

1. Training data: These models are trained on vast amounts of text data, learning to predict and generate text based on patterns and structures they observe during training. They don't possess a deep understanding of the content; instead, they rely on statistical associations. Consequently, they might generate plausible-sounding but inaccurate statements or citations.

2. Lack of fact-checking: Language models do not have an inherent mechanism to verify the accuracy of the information they generate. Their primary goal is to produce text that is coherent and contextually appropriate, which can sometimes result in incorrect or fabricated information.

3. Ambiguity and over-optimization: During training, language models may encounter ambiguous or conflicting information. They learn to optimize for generating text that appears coherent and reasonable, which can sometimes lead to them producing "safe" but false statements.

4. Bias in training data: If the training data contains incorrect or fabricated information, the model might learn to reproduce similar content. This is a reflection of the biases present in the data it was trained on.

5. Demand for specificity: When users prompt the model for specific information or details, the model may generate a response even if it doesn't have the exact knowledge. It's attempting to fulfill the user's request, which can result in the model creating nonexistent citations or false statements.

In summary, the phenomenon of "hallucination" in large language models like GPT-3 arises from the limitations of their training data, lack of fact-checking mechanisms, ambiguity in training data, biases, and the pressure to generate specific responses. These models are powerful tools for generating human-like text, but users should be aware of their limitations and always verify the information generated by them.

But don't the reasons for generating bad responses also apply to people? It seems to me the above reasons cause people to write (and believe!) things that just aren't so as well. So I asked ChatGPT to comment on that:

In your most recent response to me, you listed five reasons that Large Language Models sometimes "hallucinate" information. Humans and their writings and speech may also be subject to some or all of those five constraints that you listed which may also cause them to put forth false information. Please comment on how the five reasons you listed might also affect humans.

Here's the response:

Indeed, the five reasons listed for large language models "hallucinating" information can also affect humans in their writings and speech. Let's examine how each of these factors applies to humans:

1. Limited knowledge: Humans, like language models, possess limited knowledge based on their experiences, education, and exposure to information. When discussing a topic, they might draw on incomplete or inaccurate knowledge, leading to the dissemination of false information.

2. Lack of fact-checking: Humans may also share information without verifying its accuracy, whether due to time constraints, cognitive biases, or simply being unaware of the need to fact-check. This can result in the spread of false information, rumors, or misleading claims.

3. Ambiguity and over-optimization: Humans can encounter ambiguous or conflicting information and may form opinions based on incomplete understanding or personal biases. They might simplify complex issues, leading to oversimplified or incorrect conclusions. Additionally, people sometimes over-optimize their communication to persuade others or maintain social harmony, which may involve bending the truth.

4. Bias in sources: People are exposed to various sources of information, such as news outlets, social media, and personal networks, which may have their own biases. As a result, individuals may adopt and perpetuate these biases, consciously or unconsciously, in their speech and writings.

5. Demand for specificity: In conversations, humans may feel pressured to provide specific information or details even if they lack sufficient knowledge. To avoid appearing uninformed, they might speculate, guess, or fabricate information, potentially leading to the spread of false information.

In summary, the reasons that large language models like GPT-3 may "hallucinate" information can also be applicable to humans. It is essential to recognize these limitations, be aware of biases, and strive to verify information before sharing it with others.

So there are similarities to why both Large Language Models and people end up writing stuff that is simply and verifiably wrong. The internals are a bit different, for example AIs don't even have a concept of "appearing uninformed," but the end result is the same. Garbage in, garbage out.

 

AI Passes the Bar Exam

The latest release of GPT (GPT-4) now does quite well on many scholastic and professional tests, especially compared to its predecessor (GPT-3.5) from just a few months ago:

Blogging Again

After a two-and-a-half year break, I find my self blogging again for 2 reasons: AI tools have become fun to consider and write about and AI tools (ChatGPT in particular) have made it much faster to write about stuff. I'm finding that my first step in writing about anything is to query ChatGPT and start with its response. So far, I've explicitly identified anything written by ChatGPT in these posts. However, in the future, I may (at least) sometimes just take the response and edit it and call it a post.

I don't have readers or commenters anymore, but it's still fun to think and write. And learn to use the tools!

Artificial Intelligence and the Screechy Violin

The current set of AI tools such as ChatGPT and Dall-e have evoked a really wide range of responses. Some people are, in my opinion, somewhat over-enamored with the current capabilities of such tools, while others dismiss them out-of-hand as being inaccurate and (close to) useless.

To the latter group, I suggest considering the violin. My recent query to ChatGPT related to this topic is:

Please write a short essay comparing current, early use of AI tools to a young child learning the violin making horrible sounds, while predicting later, more sophisticated use of more sophisticated AI tools with a master violinist such as Yehudi Menuhin creating music that sounds so good that it's as if the music comes from heaven itself and is played by angels.

ChatGPT's response:

Artificial intelligence (AI) has come a long way in recent years, but we are still in the early stages of its development. Just as a young child learning the violin may struggle to produce beautiful music and create horrible sounds at first, AI tools are still limited in their capabilities and can produce errors or inaccuracies.

However, just as a young violinist can become a master musician with years of practice and refinement, AI tools have the potential to become incredibly sophisticated with further development. With more advanced algorithms, more comprehensive data sets, and more sophisticated training methods, AI tools will be able to accomplish increasingly complex tasks with greater accuracy and precision.

For example, consider the legendary violinist Yehudi Menuhin. His skill and mastery of the instrument were so great that his performances were often compared to the work of angels. In the same way, future AI tools could perform tasks with such precision and efficiency that their work appears almost magical. They could produce art, music, or writing that is indistinguishable from the work of a human expert.

However, just as a child must practice and refine their skills over time to become a master musician, AI tools must undergo rigorous testing and refinement to reach their full potential. They must be trained on vast data sets, and their algorithms must be carefully optimized to ensure that they can perform their tasks accurately and efficiently.

In conclusion, the early use of AI tools can be compared to a young child learning the violin, while the potential of more sophisticated AI tools can be likened to a master violinist like Yehudi Menuhin. With further development and refinement, AI tools have the potential to accomplish amazing feats and produce works of art that are truly awe-inspiring.

While ChatGPT's response focused on AI tools improving, I think the utility of AI tools will also depend heavily on the user. For example, I'm writing this paragraph because my query apparently wasn't quite sufficiently detailed to know to also include the improvement of the users' abilities to use the tools in the response. I guess I'm still in the category of the young child screeching on the violin. Or perhaps it's just that the violin is at this point a cheap beginner's instrument instead of a Stradivarius?

 

Digital Ghosts

Today my question for ChatGPT was:

The other day I asked you to write something in the style of Yoda and you did a good job. If everything some person wrote over a several year period was used for reinforcement learning, how closely do you think you would be able to mimic that person? And would that mimicry be a sort of "digital ghost" of that person, especially if they were dead?

And ChatGPT's answer is:

As an AI language model, I am capable of learning from large amounts of text data through a process called "training", which involves adjusting the parameters of my neural network to maximize the likelihood of generating coherent and fluent text. If a person's written work over a several-year period were used as the training data for me, I could potentially learn to mimic their writing style, including their choice of words, sentence structures, and other linguistic features.

The degree of similarity between my output and the original author's writing would depend on various factors, such as the size and quality of the training data, the specific techniques used for training and fine-tuning, and the complexity and diversity of the writing style. However, with enough high-quality training data and optimization, it's possible that I could produce text that is indistinguishable from the original author's writing, at least to some extent.

Regarding the question of whether my output would constitute a "digital ghost" of the original author, that is a philosophical and ethical question that has been debated by scholars and experts in the field of AI ethics. While I can generate text that closely resembles the original author's style and content, I don't have consciousness or agency, and my output is determined solely by the input data and the parameters of my neural network. Therefore, it's up to human judgment to determine whether such text constitutes a form of "digital resurrection" or a mere mechanical replication of the original author's work.

To me, it would indeed by a "mere mechanical replication," but so what? Let's say I had, I dunno, a favorite grandfather or something like that. And let's say that a great deal of his life for several years was recorded: what he said, what he wrote, etc. Furthermore, let's say there was a lot of video of him speaking, catching his tone of voice, his facial expressions, etc.

While ChatGPT is text based, future variants will likely have avatars and be video based. There's not a big trick to making an AI bot sound extremely similar to someone (it's already being done) and not so hard to make an avatar look like someone including facial expressions (already being done, but will be even better in the future).

Giving this hypothetical reinforcement learning to be my grandfather, some future variant of ChatGPT would enable me to sit down and have a conversation with his digital ghost! Basically forever, even though he's been dead for decades!

To me, the concept is both super cool and ultra creepy at the same time.

This reflection of my grandfather is lost for all time. But our children (grandchildren, etc.) already have a great deal of their life recorded. AI avatars of them will enable their descendants (and anyone else who is interested) to interact with these digital ghosts for eternity!

The Art of Economics: Understanding the Complexities of the Global Market

 Ok, I'm wasting a little more time with chatGPT today. The title of this post and the following are chatGPT's response to my query: 'In the style of the "Great Guys Weblog", write a post about economics.'

Greetings fellow economy enthusiasts! Today we're diving into the fascinating world of economics and exploring the intricacies that make this field so important.

Economics is the study of how societies allocate limited resources among unlimited wants and needs. It's a complex subject that touches every aspect of our lives, from the products we buy to the jobs we work. Understanding economics is crucial to making informed decisions, both as individuals and as a society.

At its core, economics is about understanding how markets work. This includes analyzing supply and demand, as well as the factors that impact them, such as government policies, technological advancements, and cultural trends. By studying economics, we can gain insights into the driving forces behind the global market and make predictions about how it will evolve in the future.

One of the key principles of economics is the concept of opportunity cost. This is the idea that there is a cost associated with every choice we make, whether it's a personal or a business decision. In other words, when we choose to spend our money on one thing, we are giving up the opportunity to spend it on something else. Understanding opportunity cost is crucial to making informed decisions, as it helps us weigh the costs and benefits of different options.

Another important aspect of economics is macroeconomics. This branch of economics focuses on the overall performance of the economy, including topics such as inflation, unemployment, and economic growth. By analyzing macroeconomic data, economists can gain insights into the health of the economy and make predictions about its future performance.

In conclusion, economics is a complex and fascinating field that touches every aspect of our lives. Whether you're an individual making personal decisions or a business owner making strategic choices, understanding economics is crucial to making informed decisions. So why not dive in and explore the art of economics today? Who knows, you might just find your new favorite subject!

So it doesn't seem to quite know "Great Guys Weblog" but still a pretty good (and enthusiastic) description of economics that I certainly agree with!

Fun with ChatGPT

Like a whole lot of other people, I've been having a lot of fun with chatGPT. Today I asked it to describe Pickleball (I'm somewhat of a fanatic) in the style of Yoda. Here's what it gave me:

Interesting game, Pickleball is. Played with paddles, it is, and a ball like a wiffle ball. On a court, small and with netting, it is played. Volleys back and forth, players attempt to score, hmmm? Fast-paced, yet strategic, Pickleball can be. Fun, it is, for young and old, hmmm?

And may the Pickleball force be with you! :-)