Want to move companies, but feel guilty

I have an opportunity to interview for a position at a new company in town that has rapidly poached about 20 of my coworkers over the last month and a half. I'm fairly certain I will get an offer, because most of the new company knows me and would probably recommend me for the position. I do enjoy most of my current work and people I work with (which was largely unaffected by the loss of so many staff members) but I'm intrigued by the opportunity to do different work that's more aligned with my education.

My problem, and question for the sub, is how to deal with guilt over leaving my current office and project work? I hate confrontation and I would dread telling my supervisor and PM about leaving just because I would get more pay. I'm pretty sure I want to leave, but I change my mind every day and I'm unhappy and anxious just thinking about it.

I know most people here will tell me to go to the new job or wherever the pay is better. I hate that I even need to have a job. I'm just a sensitive person and I'm wondering if others have dealt with a similar scenario. Cheers