Muhammad Jawad OpenAI created ChatGPT, a big language model. It is trained on a big dataset of text from the internet and is based on the transformer architecture. The model is trained to anticipate the next word in a phrase based on the words that came before it, and it utilizes this capacity to write human-like writing in response to prompts.

ChatGPT is regarded as a very strong model, capable of generating coherent and fluent language, answering queries, and even doing tasks like as translation, summarization, and more. The model can also recognize and adapt to context, making it useful for a variety of applications.

The model is now free and accessible via the OpenAI API, which allows developers to incorporate it into their applications. However, the availability of ChatGPT is unknown in the future, and OpenAI may begin charging for access to the model in the future, since the firm is focused on making income from its goods and services.

ChatGPT can assist in the generation of fresh ideas by offering prompts for certain themes and generating writings depending on the suggestions. It is crucial to remember, however, that the model is built on patterns and rules acquired from the data on which it was trained, and it may not be able to produce wholly new concepts.

Dear Muhammad Jawad , I think that ChatGPT can gives answer to your question itself.

I transfered your question to the ChatGPT: What would be future chat GPT and will it be free forever? Furthermore, how powerful ChatGPT is in terms of developing new ideas?"

After this I got the answer from ChatGPT:

"In the future, ChatGPT will likely continue to improve in terms of its ability to understand and respond to natural language input. However, it is not clear if it will always be free to use. As for developing new ideas, ChatGPT is capable of generating creative and unique content based on the information it has been trained on, but it is ultimately limited by the quality and quantity of the data it has been trained on. Additionally, it is a tool and the new ideas generated by it may not be fully thought through or feasible. It should be used as a tool for inspiration and brainstorming but not as a replacement for human creativity and critical thinking"

Just asked a logical question.

JP: Suppose all communists endorse diplomatic relations with Cuba, and further suppose Alexander endorses diplomatic relations with Cuba. Is Alexander a communist?

ChatGPT: It cannot be determined from the information given whether Alexander is a communist or not. The statement "all communists endorse diplomatic relations with Cuba" and the statement "Alexander endorses diplomatic relations with Cuba" do not necessarily imply that Alexander is a communist. It is possible for someone who is not a communist to also endorse diplomatic relations with Cuba. Therefore, additional information is needed to determine Alexander's political beliefs.

Aside from the language part, ChatGPT seemingly is able to create mental models, to manipulate them, and draw conclusions. The popular notion that ChatGPT tries to predict the next word is utterly misleading.

Regards,

Joachim

I have read the ChatGPT response provided by Vadym Slyusar and it is not the answer to this question.

The proper answer is that ChatGPT is a system composed of transformer neural networks. the GPT type design was introduced in [1] and transformers were first presented in [2]. These papers provide the background on how GPT works and is created.

As I have already answered in other threads. The transformer only has statistical knowledge of the sequence it is given. In terms of the capacity to develop new ideas , it is limited to the data it is given. Therefore it can do variations of what it is fed, but it does not create anything truly novel such as invent new technology. For more information on my criteria for such conclusion see [3]

References:

[1]Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training.

[2]Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.

[3]Geigel, A. (2022). Machine learning AI systems and the virtue of inventiveness. AI and Ethics, 1-9.

From what I understand, Arturo Geigel is very biased against ChatGPT. This is his problem. I think it's a nice tool that will make life easier for scientists, and I'll be using it in my research anyway. I think Chat GPT got a relevant answer and Arturo Geigel 's answer did nothing in the context of the interest of the author of this question.

Vadym Slyusar ,

Please cite your research on a peer reviewed journal where you prove that ChatGPT does fact checking. Please cite the references which ChatGPT used to generate your answer (assuming you did transfer learning). Please make the argument how transformer neural networks using multi headed attention can do scientific work. Tell me how the current GPT reinforcement learning guarantees that the answers are scientifically sound.

If you want to defend the technology, be ready to back it up scientifically. Use a scientific argument, since RG is a scientific forum. Then, you can criticize me on my field and I will accept it as a scientific argument and engage in it accordingly.

Further read the work that I have done in [3] and you will see that I am not against machine learning (I do work in the field every day and I lead a research team on it). The reason for me being against you using ChatGPT is in its misuse. I have no objection on judiciously using ChatGPT.

Dear Arturo Geigel , in this case, CharGPT was used quite sensibly. The answer from the ChatGPT does not contain a single stupid statement and is quite appropriate in the context of the question posed. I can subscribe to his every word.

All questions in ResearchGate are "technical questions". I don't understand your annoyance. What is the misuse of ChatGPT?

I also have publications in the field of neural networks and I know as well as you that all neural networks are just a set of mathematical formulas. However, we must face the truth - AI results are already outperforming humans in some areas, and you are trying in vain to prove otherwise.

In the context of generating a new idea, I know that the GPT chat can help provoke and already inspire the synthesis of a new idea. On the other hand, do you know about TRIZ (https://en.wikipedia.org/wiki/TRIZ)? I think that AI can give a new impetus to the implementation of this theory for the synthesis of new inventions.

I wish you more inspiration and faith in the success of your AI project.

Vadym Slyusar ,

If you know neural networks so well then you should know better. Tell me how transformers can provide a reliable answer? please I highly encourage you to do the mathematical analysis. Also please do address this scientifically.

"ChatGPT does not contain a single stupid statement and is quite appropriate in the context of the question posed"

It is not a matter of whether what comes out of it is stupid or not, it is about factual accuracy of the information that you are giving and that is supported by the literature. If you supported the answer, then I am insisting that you support the answer that ChatGPt gave with the relevant literature. You say that it is not stupid but my bar is scientific accuracy of the response (this is a research/scientific site, not LinkedIn or Facebook).

"However, we must face the truth - AI results are already outperforming humans in some areas, and you are trying in vain to prove otherwise."

This is a fallacy of generalization. If you rely on this as support for your acceptance of ChatGPT, then you have more to study than I thought.

Just because it is successful in games and sequence predictions does not make it successful in answering scientific questions. So until they do, then the answers provided by neural networks do not belong on a site where scientific questions are posed.

Dear Arturo Geigel , apparently you didn't get there. This is not a scientific site, but a forum for scientists. And read again the rubric of questions, which is called "technical questions", not "scientific" ones. You have too high expectations, but that's normal for a scientist with little experience in science, like you.

Vadym Slyusar ,

Maybe that is true, that I have little experience. Learning and gaining more experience is not a bad thing, admitting that one needs to keep learning can be taken as a good virtue to strive for. But still, you have not addressed my questions from a scientific point of view and good argumentation (without fallacies).

BTW I would have thought on telling first year college students about this, not a PhD.

To put this conversation in more context, this is someone from OpenAI itself:

'OpenAI CEO Sam Altman has publicly said that “it’s a mistake to be relying on [ChatGPT] for anything important right now.” '[1]

And what can be more important than scientific progress.

The article is worth reading as well as the technical references above from the creators of ChatGPT (ref [2] of my initial post).

[1] https://www.marketwatch.com/story/microsofts-big-move-in-ai-does-not-mean-it-will-challenge-google-in-search-11674515287?mod=home-page

Chatgpt coming out as a non-google product is what makes the changes.

Asking chatgpt how it is made will give satisfactory results. It is a nice start to AI instead of surfing the web alone Ig.

To give some of the details of ChatGPT and why it is limited, we can start with the usual transformer block diagram[1]. What is worth highlighting is the input stages. These are the input embedding, positional encoding and the multi head attention. The input embedding can be something like CBOW[3] which generates a multidimensional space where vectors that are related can be clustered closely. This would be the equivalent of synonymy. The positional encoding and the multi head attention directs the NN towards what in the sentences are important. This would be equivalent as enforcing syntax rules and similar words on the vector space can be exchanged with one another. What can be seen as amazing is that it can enforce some semantic rules as well. Upon closer look this is not something new nor novel. If you look at Verbnet[4] you can see that verbs are tied to a syntactic enforcement rule. Given that you focus on teaching a transformer the correct usage of a particular verb it can enforce the syntactic rules such as those of Verbnet. Further, given a narrow interpretation data set, the CBOW encoding will then use the words clustered together to substitute those in the examples.

One of the restrictions on these architectures is that syntactic enforcement is not equivalent to causal enforcement. Given abroad enough data set it can conclude that the 'earth is flat' or that 'we never went to space'. This factual checking depends on purging from the data set the corresponding sentences that could lead the algorithm to output such sentence. But, given the sale of the data that is being fed into these architectures, the fact checking of all possible outputs is not feasible. Further, the semantic relationships drawn from embedding and positional encoding do not necessarily reflect the intended meaning of particular instances.

One promising area that can help (but it is not there yet, is causal machine learning[5]. The limitation in these algorithms is that they are still in their infancy and I do not see ability to scale to keep up with the scaling done for syntactic and shallow semantic enforcement.

References:

[1] image from Wikipedia and it is the same as ref 2

[2]Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.

[3]Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.

[4] Schuler, K. K. (2005). VerbNet: A broad-coverage, comprehensive verb lexicon. University of Pennsylvania.

[5]Peters, J., Janzing, D., & Schölkopf, B. (2017). Elements of causal inference: foundations and learning algorithms (p. 288). The MIT Press.

While the previous post of mine focused on creative content, I want to focus on a claim made by some in the ML community about semantic understanding by neural networks such as ChatGPT.

I traced the origin of the claims that transformers can encode semantic information which can be found in [1]. The problem that I find (and disturbing at that) is that as stated by the author:

"This work is intended to be exploratory. We focus on one particular encoder–BERT–to explore how information can be organized in a deep language model, and further work is required to determine to what extent the trends hold in general"[1].

This caveat seems largely unnoticed in claims of semantic understanding. Further, while there are metrics in this work which shows a pattern, they are not correlation studies nor do they prove causal relationships that can be traced to semantic theories.

Further in [1] it states "Furthermore, our work carries the limitations of all inspection-based probing: the fact that a linguistic pattern is not observed by our probing classifier does not guarantee that it is not there, and the observation of a pattern does not tell us how it is used. For this reason, we emphasize the importance of combining structural analysis with behavioral studies". The same can be put in reverse. Just because there may seem to be a linguistic pattern does not guarantee that it is there. Before making such claims a pattern must be linked to a particular variable and then show the causal link between phenomena and the variable.

One of the underlying assumptions of the ML community in stating that NNs can have semantic understanding falls on the distributional hypothesis (yes hypothesis, not law)[2]. What I find very significant about this work is the author's analysis of Zellig Harris where he states:

"is a strong methodological claim with a weak semantic foundation. It states that differences of meaning correlate with differences of distribution, but it neither specifies what kind of distributional information we should look for, nor what kind of meaning differences it mediates".

This to me seems to be the crux of the problems on claims about neural networks "understanding" semantics.

If anyone sees a further elaboration of [1] that can shed more structured evidence to revise my stance, let me know.

References

[1] Tenney, I., Das, D., & Pavlick, E. (2019). BERT rediscovers the classical NLP pipeline. arXiv preprint arXiv:1905.05950.

[2] Sahlgren, M. (2008). The distributional hypothesis. Italian Journal of Disability Studies, 20, 33-53.

ChatGPT is an artificial-intelligence (AI) chatbot developed by OpenAI and launched in November 2022. It is built on top of OpenAI's GPT-3.5 and GPT-4 families of large language models (LLMs) and has been fine-tuned (an approach to transfer learning) using both supervised and reinforcement learning techniques.

Dear respected researchers

We have worked on a paper on that particular issue (ChatGPT and Academic research)

The article's link:

https://journals.cspc.edu.ph/index.php/jemds/article/view/175

I hope you will enjoy the article and it will be helpful for academic people.

Regards.