Business owners that do their own sales, how do you keep track of your conversations if a prospect calls you back and you're not in front of your CRM?

So for my business, a lot of what I do is related to phone calls, emails, and networking.

Often times, I will call on anywhere from 20 - 50 prospects a day, often related to different business offerings depending on what industry they are in and intent data.

I have my own CRM, so if they call me while I'm at my desk I can look them up and check my notes.

However, sometimes I will get a call back and I'm not in front of my computer. Sometimes on these calls they will just say something like "I missed a call from this number" and won't tell me who they are.

I don't want to come off as absent minded by saying something like "I'm sorry, can you remind me your name, your industry, and what company you are calling from?" Because I don't want it to look like I'm just calling random people, I'm not, I just can't remember everyone off the top of my head. And I don't want to just guess since for one industry it could be something like consulting and for the other it could be software, so I don't want to give the wrong pitch for something that doesn't make sense.

Is there a way to ask for someone to clarify who they are, their industry, and the company name that still sounds professional and not like I'm a small fry that can't remember everyone they called?