Hamlet is engaging in the double meanings of the words “sound” and “compass”. You can make a sound with a musical instrument from its lowest note to the top of its range (compass). But sounds can also mean to probe, and compass can mean cunning; Guildenstern is probing Hamlet to find out why he’s being so cunning.