In the world I am building, I want some of the giants to be a good bit bigger than other giants but the smaller ones to still be giant to the average human and the bigger ones to not be too big.
How can I make this work without making them to big all together?
They are different races of giant.
A crowd of people in a big American city can be fairly varied. People have ancestors from varying regions of the world. These ancestors looked different and were culturally different from one place to the next. In the new city, their descendants live together, but have not been together long enough to breed to homogeneity. You can tell where their ancestors were from by how they looked.
So too your giants. This would also be fun to write because size would correspond with other traits. Your frost giants, for example, would be the tallest and tend to wear their hair braided through hollow rocks, with additional rock jewelry and also very bad breath due to their diet. The fire giants would be the smallest; intense, ruddy skinned, shaved, tattooed; they smell like burnt toast and cloves. Sea giants are between in size; mellow and blue skinned, a little chunky in the manner of a sea mammal, prone to long inscrutable gazes and communication via extemporized song.
The giants live and work together but as with humans, cultural differences might sometimes make things difficult. For example the fire giants stereotype the frost giants as smelly, clumsy and for the latter reason dangerous. Some young frost giants might try to be more like the fire giants, shaving and trying to take care of their teeth.
If you want your giants to be a separate species related to Homo sapiens or a subspecies of Homo sapiens it is quite possible that they will have a large range in size.
I once was slightly acquainted with a boy who was about five feet (152.4 centimeters) tall and looked like a cuter than average 12-year-old. One time at a restaurant he was one of the wait staff and one of my companions asked how old he was and he said 16. Although she didn't believe him, he should have been 16 to work in a place that sold liquor, and he should have had to produce proof of his age to get hired.
I don't know how old the tallest boy I ever saw was, but he would have been a lot taller than five feet (152.4 centimeters) when he was 16, whether he was 16 before or after I saw him. I was about 5 feet 10 inches (177.7999 centimeters) to 6 feet 0 inches (182.88 centimeters) tall, and when he stood near me he was clearly several inches taller, perhaps about 6 feet 2 inches (187.9 to six feet 5 inches (195.5 centimeters). Though he was bigger than most men, his face looked quite young and childlike.
So I have seen people who were about 10 or 20 percent taller or shorter than the average for their age.
While newborns vary in size and shape as much as adults do, full-term babies tend, on average, to weigh between 5 pounds, 11 ounces and 8 pounds, 6 ounces. They're usually between 19 and 21 inches long, with a head circumference of about 13 1/2 inches.
https://www.parents.com/baby/development/physical/charting-babys-growth/1
Newborn babies can't stand, but if you hold them upright their height should be about the same as their length lying down.
The shortest man ever measured, Chandra Bahadur Dangi (1939-2015), was 54.6 centimeters or 21.5 inches tall.
The shortest woman ever measured, Pauline Musters (1876-1895), was 61 centimeters or 24 inches tall.
The shortest man alive, Junray Balawing (b. 1996), is 59.9 centimeters or 22.4 inches tall.
The shortest woman alive, Jyoti Amge (b. 1993), is 62.8 centimeters or 24.7 inches tall.
Considering how little larger than normal babies they are as adults, it is easy to imagine that such short adults were probably much smaller babies than normal ones.
The shortest baby ever, Nisa Juarez, (b. 2002) was 24 centimeters or 9.4 inches tall, and weighed only 320 grams or 11.3 ounces when she was born 108 days prematurely.
https://en.wikipedia.org/wiki/List_of_the_verified_shortest_people2
The tallest man ever recorded, Robert Wadlow (1918-1940), was 272 centimeters or 8 feet 11.1 inches tall.
The tallest woman ever recorded, Zeng Jinlian (1964-1982), was 248 centimeters or 8 feet 2.75 inches tall.
The tallest man alive, Sultan Kosen (b. 1982), is 251 centimeters or 8 feet 2.8 inches tall.
The tallest woman alive, Sun Fang (b. 1972), is 221 centimeters or 7 feet 3 inches tall.
And of course the prehistoric "Giant of Castelnau" was estimated from bone fragments to be 350 centimeters or 11 feet six inches tall.
https://en.wikipedia.org/wiki/List_of_tallest_people3
So if the tallest and shortest adult humans alive today are brought together for some event one would be 4.1903 times as tall as the other, while if the tallest and shortest of all time were together one would be 4.98 times as tall as the other.
Of course children who are not much taller than newborn babies can be seen walking around with adults. The tallest man ever was 11.3333 times as tall as the smallest living premature baby. As a teenager, Francis Joseph Flynn (1864-1898) was 53.3 centimeters or 21.0 inches tall, and he could walk, and the tallest man ever would be 5.103 times as tall as him.
Of course these examples of the shortest and tallest humans come from a time when billions of specimens of Homo sapiens are alive, and your population of giants is likely to be much smaller than billions.
So your giant population could very well have adult giants who are about 10 percent taller or shorter than the average adult of their gender without having a population of billions of giants. If the average height of a giant man was 10 feet (304 centimeters), giant men might range from 9 feet (274.32 centimeters) to 11 feet (335.28 centimeters) in height.
If the average height of a giant man was eight feet or 243.84 centimeters, and their heights could vary from the average by 20 percent, giant men would be between 6.4 feet or 195.072 centimeters and 9.6 feet or 292.608 centimeters. In that case the height range of giant men would overlap with that of human men, and if their appearance was human enough, short giants might be mistaken for tall men.
In any case, giant children might be mistaken for human children years older than them, if they look human enough.
