Most of the culture of ancient Rome derived from the older, highly advanced Greek civilization. Roman intellectuals learned Greek and studied in the sophisticated city of Athens. Cicero has just made a disparaging comment about Caesar, which could get him in trouble if overheard by the wrong person. He spoke it in Greek so that only his intellectual friends would understand, those that "smiled at one another and shook their heads."  

On a related note, because medieval European culture descended directly from that of the Roman empire, a public education in Shakespeare's day was devoted almost entirely to mastering Latin and studying the ancient Roman writers. Every educated person knew Latin, but just as in Cicero's Rome, only a few intellectuals knew Greek. As a result, this expression, "It's Greek to me," was already common in Shakespeare's day.