If you retire too early, "you're going to pay for it for the rest of your life," warns Jim Cramer, host of CNBC's "Mad Money" and Investing Club.

That's because people who retire decades early will probably need "a lot more" money saved up than they think, Cramer tells CNBC Make It. "A lot of people are betting against themselves. They're betting against how long they'll live."

Relatedly, Cramer is skeptical of the movement known as FIRE, which stands for "Financial Independence, Retire Early." With FIRE, the goal is to put as much income into investments as possible — somewhere between 50% and 75% of your income — so that you can retire as early as your 30s or 40s.

"I talk to a lot of people who want to retire early," says Cramer. "They don't want to leave the earth without doing amazing things. They want to climb Machu Picchu. They want to see the pyramids. Maybe they want to climb Everest."

However, such a lifestyle is expensive and can easily be derailed by any number of expenses that can happen over decades of retirement, Cramer says.

For example, you might end up owning a nice house that you can't afford, want to travel more than you originally planned or develop a medical issue not covered by your health insurance. Although Social Security and Medicare will provide some financial help in retirement, those programs won't necessarily cover all of your unexpected medical debt, if you have it, which can sometimes cost tens of thousands of dollars.

The bottom line is that you never truly know how much you'll need — especially if you plan to be retired for more than 30 years, Cramer says. 

Your needs also change as you age, he says. You might be dead set on early retirement now, only to decide later that you want to spend money on things that improve your quality of life, such as trips to see family or gifts to your children.

By retiring too soon, you might not have the flexibility to be able to increase withdrawals from your retirement funds later.

"A lot of people make a judgment. And the judgment is, 'I'll always be able to afford'" retirement, he says. But for many early retirees, "you won't."

To learn more about investing, you can join the CNBC Investing Club with Jim Cramer at a discounted rate.

Sign up now: Get smarter about your money and career with our weekly newsletter

Don't miss: Jim Cramer invested $100 a month in his 20s—even while living in his car: 'I put that money away, and it made me a millionaire'