When I woke up this morning, I noticed that the GPS icon was showing on my iPad. The only app I had running was my alarm clock, which doesn’t use Location Services. It was odd but didn’t give it too much more thought. Then I was browsing Twitter and noticed someone had tweeted that the new Google Search app (with Google Now) had a flaw, leading to the GPS being always on. Aha! Perhaps mystery solved. I checked my iPhone. Sure enough. GPS icon active.
Some confusion going on here about the 'GPS icon' on the iPhone. The iPhone (and iPad) has a small icon in the menu bar that appears when something on your phone is using Location Services. Location Services is an API that is used to identify where you are.
I would guess that the confusion comes from the fact that the icon is the same one used in the Maps app to say where you are (ie. to access Location Services), which is a concept that everyone is familiar with through GPS devices – and which uses GPS on devices that have a GPS chip. (Wifi-only iPads and iPod Touch have Location Services, but no GPS chip.)
But "Location services" doesn't mean GPS. GPS is a part of it, but at any given time, a phone that is connected to a network has some information about where it is, because it knows which network cells it is currently connected to. Similarly, devices on a WiFi hotspot can deduce information about where they are based on the hotspot and IP address. Location Services allows an application to tap into this information, and the icon is there to let you know that something you use is tracking where you are. If more detail is needed, then Apple's APIs will then use services like GPS to get more accurate locational data (at different levels of accuracy, depending on what is needed— more accuracy requires more power.)
This might mean that an application will 'wake up' when you are in a certain area and then switch on GPS to confirm (for example) whether you are actually in the area (say, a train station, a shop, at home) or just nearby.
Apple's developer documentation explains how this all works on a technical level, but the bottom line is that the location services icon is not a GPS icon, and it doesn't mean that something is using GPS and draining your battery.
Whether you still want to use an application that is constantly tracking your location is up to you though. Of course, a side effect of this is that even if you trust Google with that information, you won't be able to easily see if any other applications are using Location Services without going into the Settings menu to manually check.