We would have to do more things by trial and error. We'd have to build things as we figure out how to build them. We may not be able to build just anything we please. Mathematics gives us the power to picture something, and build it entirely in our heads (or perhaps in the computer), and then build it in one big effort from start to finish.
Grand architecture would thus have more "corrections" in it. You'd see structures whose design permits the builder to make very subtle adjustments to the structure to make up for oversights in lower levels which calculus would have caught. This would lend itself to a much more organic structure, where we spend less effort demonstrating how proudly we can trust a steel girder to support our weight, and more effort ensuring we never really have to trust the girder in the first place. I can see an architect bouncing up and down lightly on his cantilever, and making the decision to adjust the building on top because the cantilever isn't as strong as he or she might hope.
I'd say skyscrapers would be unlikely. I'm confident that, at some point, we could manage to construct them as architecture evolves. However, we also have to consider that a lack of math would affect much of the rest of our life, and we may not have the desire to push steel and glass structures into the sky.
What might be interesting is to look at what vessels can act like math in the absence of higher math. I think it'd be interesting to see a world where "plans" for a building are actually stored in a song sung by the foreman as the laborers work their own tune with the material.