It depends on where you are. Some jurisdictions will cite (and even convict) for holding phones, while some look for actual phone use (holding up to ear) to cite.
For what it's worth, using the phone GPS and Contacts applications are not illegal, so touching phone itself is not illegal. You still can be cited, but it's up to you to defend yourself against the officer's claim in court. People have had their tickets dismissed based on GPS use arguments. However if you were actually talking on the phone and the officer can testify seeing you talking on the phone, you don't have much of a case.
Don't argue with cdwjava. While many here offer advices based on personal feelings and assumptions, his advices are generally good and based on experience. You many not like the answer, but they are fair.

