What penalties and interest does the IRS charge?

There are 3 penalties.
Paying late

If you file a return but don't pay all tax owed on time, you'll have to pay a late payment penalty. The failure-to-pay penalty, as it is called, is one-half of one percent (0.5%) on any payments made after April 15 for each month, or part of a month, up to a maximum of 25%. Note: The one-half of one percent rate increases to one percent if the tax remains unpaid 10 days after the IRS issues a notice of intent to levy property. Also read IRS took money out of bank or paycheck

If you file your return by its due date and request an installment agreement, the one-half of one percent rate decreases to one-quarter of one percent for any month in which an installment agreement is in effect.

Filing late

If you owe tax and don't file on time (that includes extensions) there's a penalty for not filing on time. The failure-to-file penalty is five percent (5.0%) of the tax owed for each month, or part of a month that your return is late, up to a maximum of 25%. If your return is over 60 days late, there's a minimum penalty for filing late: It's $435 100 percent of the tax owed, whichever is less.

Underpayment of estimated tax

Currently the estimated tax penalty is 4.0% figured on the underpayment amount that should have been paid on each of the 4 estimated tax payment dates and computed for the number of days each payment is late.

Interest

 

Interest accrues on any unpaid tax from the due date of the return until the date of payment in full. The interest rate is determined quarterly and compounds daily. You an find a chart of historical rates here.  

Note that interest is also charged on unpaid penalties.

How payments are applied

The IRS applies payments to the unpaid tax first, then to penalties, and then to interest. Any penalty amount that appears on your bill is generally the total amount of the penalty up to the date of the notice, not the penalty amount charged each month.

Also read The IRS Has Charged Me Penalties and Interest. Can I Get Them Removed?