It's extremely tough to detect many space objects and more often or not, what we do detect is from a mix of luck and tenacity. It's important to remember the size of space and that most of us hold a pretty distorted view of it...if you've ever seen a picture of the Sun and Pluto on the same piece of paper, then you've witnessed this distortion. If the sun was reduced to the size of a pea, by scale, Pluto would be a microscopic (not visible to human eye) spec of dust some 35 feet away. Scale representation of Earth wouldn't be on the same piece of paper either.
If we assume the spacecraft has taken steps to mask it's deceleration and isn't giving off any light or radiation itself, lets compare what it's taken to detect objects of smaller sizes from Earth:
In 2011, we discovered Pluto has yet another moon. https://www.nasa.gov/mission_pages/hubble/science/new-pluto-moon.html This moon 6 to 15 miles across and was discovered after a decently intense picture set was taken by the hubble telescope attempting to find out what the new horizons spacecraft would have to deal with. There are two component that were needed to find this planet...first we have to take the picture and second we need to have someone look at that picture and recognize what they are looking at.
In 2014, we discovered an asteroid between Saturn and Uranus's orbit had rings. This was entirely due to an event known as occultation, which is when an object passes by a distant star and we can see the object as a shadow in the distant star's light. https://www.space.com/25225-asteroid-rings-discovery-video-images.html It took 7 telescopes all pointed at the same object at the same time to successfully detect these rings.
As you can see, we need to focus many resources in a known area to detect something in the outer solar system. If we were going to locate it via telescopes, we would need 3 things to happen at the same time : The ship would have to traverse across the light making it to Earth from a distant star, we would need multiple telescopes pointed at that star, and we would need people actively looking at those results. Anything could happen, but this pushes past dumb luck into accidentally finding a needle in a New York sized haystack.
**Just to throw it out there, once again in the dumb luck category, but we do have a few recon spacecraft in the outer system. Not likely, but one of these crafts spotting and sending back images of an alien craft would probably be much more likely than we would be spotting it from Earth.
I'd be hard pressed to state that we'd find a 50km wide craft in the outer solar system without some extremely lucky circumstances occurring. When we get to the inner solar system, we are a bit better at detecting these bodies, but only a bit. Lets say we have a pretty good history of missing things.
http://earthsky.org/space/asteroid-2017-oo1-close-pass-undetected An object between 25 and 78 meters passed between the Earth and the moon, and we detected it 2 days after it's closest point to Earth. I don't know how large your ship is, but under 100 meters could likely land on Earth before we detected it.
More recently, an asteroid passed us that appears to be an interstellar asteroid. https://news.nationalgeographic.com/2017/10/interstellar-solar-system-asteroid-comet-space-science/ This several hundred foot object was sighted 5 days after the objects closest point with Earth and was already on it's way back out of the solar system. If your alien craft is moving at speeds required to exit the solar system, I doubt we'd detect it until after it was here.
Infact, our near earth asteroid detection has been painfully slow. http://www.cbc.ca/news/technology/nasa-near-earth-objects-asteroid-detection-program-slammed-by-audit-1.2767688
"NASA estimates that it has identified only about 10 per cent of all asteroids 140 metres and larger," Martin wrote. "Given its current pace and resources, (NASA) has stated that it will not meet the goal of identifying 90 per cent of such objects by 2020."
We have discovered upwards of 90% of all objects over 1km diameter, but this has taken several year to go over. Unless this spacecraft is multiple KM in diameter, we wouldn't stand much of a chance in discovering it what-so-ever.
If we go a bit bigger, lets take Shoemaker-Levy 9 as an example. From estimates, this 1 to 10km large object was captured by Jupiter somewhere in the 60's or 70's before it's final impact into Jupiter. We discovered it completely accidentally while looking for Near Earth objects in 1993 long after it fragmented.
We missed a 1km one that impacted Jupiter in 2009 and have to guess at it now. https://en.wikipedia.org/wiki/2009_Jupiter_impact_event This wiki page includes a note on visibility:
Assuming it was an inactive comet (or asteroid) about 1 km in diameter, this object would have been no brighter than about apparent magnitude 25.[17] (Jupiter shines about 130 billion times brighter than a 25th magnitude object.)[18] Most asteroid surveys which use a wide field of view do not see fainter than about magnitude 22 (which is 16x brighter than magnitude 25).[17] Even detecting satellites less than 10 km in diameter orbiting Jupiter is difficult and requires some of the best telescopes in the world.[19] It is only since 1999 with the discovery of Callirrhoe that astronomers have been able to discover many of Jupiter's smallest moons.[20]
Unless extremely lucky, we probably wouldn't notice a 10km wide ship until it took up orbit around the moon.
Additional note on visibility:
http://www.icq.eps.harvard.edu/MagScale.html
It might actually be easier to determine what visibility rating your spacecraft would have and note from there what it would look like to us on Earth. We detected a magnitude 15.3 comet just outside of the orbit of Mars recently. https://www.universetoday.com/136564/new-comet-c2017-o1-asas-sn-takes-earth-surprise/ I'd be curious how big a space ship would have to be / what distance it would have to be to be 15.3 on this scale...this comet is actively giving off a decent amount of light that makes detecting it possible.
edit:
Might as well throw out the absolutely huge option. If this was a death star like alien ship, I'd consider it emerging into the solar system from the OORT cloud as when we would start having a chance of detecting it. Remember, we've only identified dwarf planets about half the size of Pluto out in the oort cloud in 2015 https://www.space.com/31100-most-distant-dwarf-planet-found.html and are just barely starting to see into the inner OORT cloud let alone the outer.
too broad
. This question is an idea generation question that requires broad assumptions by answerers (alien ship size/composition/tech, human tech level). – Azuaron Nov 29 '17 at 20:32