“In March 2012 I contributed 5000 to my Roth for the 2011 tax year. Turns out, due to my AGI, I was not allowed to make any contribution. I only just realized this is the case and have missed the extended deadline for penalty free withdrawal (Oct 2012 I think).”=====as you contributed too much money, you must pay a 6 percent tax penalty on the money if you fail to withdraw it before your income tax due date, including extentin, Oct 15 th 2012. When you take the withdrawal to correct your contributions, you must also withdraw any earnings on the excess contribution.
Last week I withdrew the excess (5000) but I gather I will have to pay a 6% penalty (300) for both 2011 & 2012. The value of the Roth has actually fallen slightly over the 2 year period so there are no earnings to speak of.”===== If you didn't correct the excess contribution for 2011, you had to pay $300 excess contribution tax . And if you left the problem uncorrected beyond the end of 2012, you owe another $300. You'll continue to owe this tax each year until you correct the excess contribution.
“My question is how do I report all this to the IRS? From reading various websites it seems that I need to file form 5329. But for which tax year (2011 or 2012 or both)?”====== it depends; it is either 2012 or 2012 or both 2011 and 2012. You need to record the distributed earnings on your tax return as gross income on Line 15, Form 1040.ALSO you need to complete Form 5329 to determine penalties. If you are under 59 1/2 years of age, Part I determines the 10 percent tax penalty on earnings. Part IV determines the 6 percent penalty owed if the excess was not taken out by the deadline.
“ Do I need to file an amendment to my 2011 return? I will not receive a 1099-R until Feb 2013, so I'm not sure if I have to wait until then to take care of this.”== This is a 2011 excessive contribution; you will need to file an amended 2011 return to pay the penalty for an excess contribution since it was not returned within 6 months of the due date of the return. This has nothing to do with the earnings that the excessive contribution earned while in the IRA, but is a separate 6% excise tax that applies to any excess contribution to a Roth IRA for EACH YEAR that excess remains in the IRA. Second, you will also need to figure the same penalty for 2012 since the penalty applies for every year the excess amount remains in the IRA (It can be reduced by the amount that you were allowed to contribute to the Roth in 2012- if any). (If you do not do that, you will be getting another letter from the IRS next year to amend your 2012 tax return). date plus 6 months, but would not apply if the earning were withdrawn two years later)