The left and the right don’t agree on much, but increasingly they agree on one thing: Mitt Romney is completely failing to address public concerns that he may have a shady personal and professional history of tax avoidance. (Romney refuses to release more than his returns from 2010 and 2011.) Liberals and conservatives have different reasons for taking this view.

Liberals believe that Romney’s established history of using foreign bank accounts to shelter money and bet against the dollar, as well as his bizarre history of being paid by Bain Capital long after he supposedly stopped working there, raises valid questions about Romney’s values and potential conflicts of interest. Therefore, the public has a right to know what else is in Romney’s financial history. How much lower has his tax rate been than of the average working stiff? How much does he give to charity? How did he amass a preposterously large IRA?

Conservatives, on the other hand, are mostly just concerned that Romney is making a political error. He already is perceived by much of the public as an out-of-touch fatcat. Every day Romney is forced to play defense on his secretiveness is a day he spends reminding people that he is far richer than they are and still pays a lower tax rate, since his income is mostly capital gains. He, and his supporters, would rather he was talking about the high rate of unemployment. If Romney is going to ultimately cave on his taxes anyway, he would do better to get them out now, while swing voters are not yet paying close attention to the campaign, than in September or October.

Presidential candidates always release several years of tax returns. Romney’s father George set the modern precedent, with twelve years, in 1968. On Thursday morning ABC’s Robin Roberts asked Ann Romney—noting that Obama released seven years of returns and both Bushes at least ten—why her husband won’t follow the bipartisan tradition. Romney offered at first the completely irrelevant talking point that they tithe to their church and that Mitt chose to forgo a salary as governor of Massachusetts. When pressed by Curry, she admitted that they fear there will be fodder for attacks in their tax history.

Romney’s own personal history on the subject is mixed. He never released any returns in his previous campaigns. Bain sources told the Huffington Post that he did not think he would have to in a presidential run and wouldn’t have run for president if he thought he would.

It’s unclear why he is so afraid, since he gave twenty-three years of returns to the McCain campaign when he was being vetted for vice president. McCain says there was nothing in Romey’s taxes that disqualified him from being chosen as running mate. On the other hand, as TPM’s Brian Beutler points out, former top McCain advisor Steve Schmidt said on MSNBC that it is not worth the cost to Romney to give up the returns. Schmidt, of course, would know better than anyone else what was in them.

But Schmidt’s opinion is not that widely held, even on the right. Much of the Republican and conservative establishment is calling on Romney to release his returns, from former RNC Chair Haley Barbour to Texas Governor Rick Perry. The most respected conservative pundits, from National Review to Washington Post columnist George Will are saying the same. Think Progress compiled a list of twenty prominent Republicans who have urged Romney to do so.

It is notable how badly Romney has handled this whole issue, and how bizarrely unprepared for it he has seemed. During the primaries he was reluctant to release any returns, gave wishy-washy answers as to what he would reveal and when, and then gave up his 2010 return after being pressured by his opponents and the media. This only reinforced his reputation as a slippery politician. In an editorial this week, National Review debunked Romney’s pathetically unconvincing justifications for not releasing his returns, and made the campaign strategy argument for getting it over with:

The Romney campaign says he has released as many returns as candidate John Kerry did in 2004, and cites Teresa Heinz Kerry’s refusal to release any of her tax returns. Neither is an apt comparison. John Kerry actually released returns from 1999 through 2003, and also released tax returns during his Senate runs. As for Teresa Heinz, Romney isn’t the wealthy spouse of a candidate, but the candidate himself. In 2008, John McCain released two years of returns, but he had been filling out financial disclosure forms for decades as a senator. Romney protests that he is not legally obliged to release any tax returns. Of course not. He is no longer in the realm of the private sector, though, where he can comply with the letter of the law with the Securities and Exchange Commission and leave it at that. Perceptions matter.

Romney may feel impatience with requirements that the political culture imposes on a presidential candidate that he feels are pointless (and inconvenient). But he’s a politician running for the highest office in the land, and his current posture is probably unsustainable. In all likelihood, he won’t be able to maintain a position that looks secretive and is a departure from campaign conventions. The only question is whether he releases more returns now, or later — after playing more defense on the issue and sustaining more hits. There will surely be a press feeding frenzy over new returns, but better to weather it in the middle of July.

As The Daily Beast’s Andrew Sullivan notes, “This is such obvious advice that the real question becomes: what on earth could be so damaging that Romney would risk this agonizing twist in the wind? Did he pay no taxes at all in 2009?” The Romney campaign denied that on Wednesday, but in such a way that it’s still possible Romney only paid, say, 3 percent one year.

There is good reason for Republicans to fear swing voters’ response to Romney’s stubbornness. Public Policy Polling finds that 56 percent of Americans and 61 percent of independents think Romney should release his tax returns from the last twelve years, while only 34 percent of Americans and 27 percent of independents think he should not.

Barbour is not the only former RNC chair to call on Romney to release his returns. On Tuesday night Michael Steele and Nation editor Katrina vanden Heuvel appeared on The Ed Show and they agreed that Romney owes it to the public to follow the example set by his father. Sooner or later, Romney will come to accept this, or else he will hand Obama an issue with which to criticize him although the way to Election day.

Visit for breaking news, world news, and news about the economy