On the death of the IRA owner, in this case your Mother, the value of her IRA account would have been includable into her taxable estate for estate tax purposes and any payments from the IRA to you or your brother are subject to income taxes. This is based on the theory "that no income taxes were paid during the life of the original owner."
The periodic payments are taxed under the rules of § 72 for annuity payments, as modified by I.R.C. §408(d)(2). Distributions to other beneficiaries are taxed in the same manner as distributions to the original IRA owner. Thus to answer your question, "Should we have been exempted from paying taxes on this as it was part of the estate?", the answer is No! You would have to pay income taxes on the IRA distribution received from your Mother Estate.
Remember, the monies that are in the IRA have never been subject to any Income taxes and so these amount upon distribution to you would be taxed to you as ordinary income. The estate exemption that you are referring to is either to the Marital Exemption or By-Pass Trust. None of these exemptions are applicable in this case. Regretably, I think you are liable to pay taxes on these IRA distributions.