Catch something that is not a fish

Is there some trick to this? I've been fishing for almost an hour now and just catching fish.

Edit: Sardeliac's suggestion of fishing without bait off a shore in shallow water 100% worked for me. Keeping this up for anyone else who needs it.