Temperature will vary and weather will change. Even if we consider 20/30/40/50 years definitely there will be changes in the atmosphere as its natural. So why we did not use yearly weather or anything else other than climate change??
The year to year fluctuation/changes/variability is what we call "Climate Variability". I will recommend this textbook for you to gain the basic insight.
"Climate Change Science: A Modern Synthesis: Volume 1 - The Physical Climate 2013th Edition "
Climate change, as the name indicates is a term commonly used to show a consistent change in the weather conditions of a place over a relatively long period of time. With climate change data, we can show the trend and magnitude of climate variations within a time framework. Yearly weather on the other hand can help us to make comparisons.
The set of definitions given below by Keely Dinse (Source: https://eos.ucs.uri.edu/EOS_Linked_Documents/michu/michug11017.pdf ) is informative
Climate variability – The way climate fluctuates yearly above or below a long-term average value.
Climate change – Long-term continuous change (increase or decrease) to average weather conditions or the range of weather.
Climatological normal – 30-year average of a weather variable.
It seems you are mixing climate variability with climate change.
While the climate tends to change quite slowly, that doesn’t mean we don’t experience shorter-term fluctuations on seasonal or multi-seasonal time scales. There are many things that can cause temperature, for example, to fluctuate around the average without causing the long-term average itself to change. This phenomenon is climate variability, and when scientists talk about it they are usually referring to time periods ranging from months to as many as 30 years.
For the most part, when discussing climate variability, we’re describing natural (that is, non-man-made) processes that affect the atmosphere. For example, the North Atlantic Oscillation (NAO) refers to anomalous changes in atmospheric pressure at sea level that occur near Iceland and the Azores High. NAO-positive phases are often associated with above-average storm counts over parts of Europe and the U.S. You’re also likely familiar with the El Niño Southern Oscillation (ENSO) phenomenon near the equatorial Pacific Ocean, where fluctuations of sea surface temperatures typically alternate every few years between a warming phase (El Niño) and cooling periods (La Niña), with a neutral phase in between. Many researchers have found that negative ENSO years are correlated with a higher probability of Atlantic hurricane formation, as well as warmer, dryer weather in northern states.
Alterations to the earth’s atmosphere that occur over much longer periods—decades to millennia—are characterized as “climate change.” While climate change can be caused by natural processes—such as volcanic activity, solar variability, plate tectonics, or shifts in the Earth’s orbit—we are usually referring to changes attributable to human activity when talking about climate change, such as increased greenhouse gas emissions. The latest (Fifth) Assessment Report from the Intergovernmental Panel on Climate Change (IPCC 2013), for example, found that on average global temperatures increased about 0.85°C from 1880 to 2012, and concluded that more than half of the observed increase in global average temperatures was caused by elevated emissions of carbon dioxide and other greenhouse gases.