I think the point Crispin was trying to make is that 'serious survey research' is also completely beholden to those who opt to have their say. There is no way of making people take part and in fact market research seriously annoys many of the people it is done to. I am not sure how representative those who agree to take part could ever be.

The ability to show that the sample comprises people with different ages, incomes etc is false comfort as these factors fail to recognise the different values people hold.

In my view market research and community engagement can compliment each other but the danger is when people try to pass one off as the other.

Just for the record the voting mechanism on Bang the Table sites is optional and can be switched off by clients according to their wishes and needs. However, I agree that care always needs to be taken when interpreting results.

I think your second point is so crucial, that is what we miss in those who don't participate. When Fishkin ran his Tomorrow's Europe Deliberative Poll, I wrote a scathing post just on that point. However, preliminary work by John Dryzek on our CP is perhaps surprisingly not showing a strong communitarian skew to the participants who accepted the invitation. And there were still plenty of cynics amongst participants, which is probably a good thing.

Has there been any research surveying participants after an event to test the hypothesis that beliefs and values tend to cluster within age/sex cohorts and/or that the sample demographics reflected other values based strata like religious affliations, ethnicity, education and voting preferences? It would be interesting to know whether there is any sample skewing in terms of the willingness to participate brought about by the socio-cultural background of the participants.

Crispin

I don't think a deliberative event should be presented as an experiment, research exercise or a newfangled focus group. While the organiser (which may include academics, okay) is there to design and facilitate a conversation amongst participants, its purpose must be to deliver a recommendation to an authority like a govt dept. Any conversation with a researcher is a distraction!

SRS works in two stages. First, a lot of random invitations are sent out, and a short survey is included. They tick off gender, age, etc. Then the second stage is the stratified random selection from those who accept the invitation, based on their survey submissions.

The problem of asking them questions about their beliefs (eg. who they voted for, social capital links, priority between individual or community pursuits) is that cynics would not believe that the selection is random!! I can reveal that many at the Citizens' Parliament thought that SRS is a fancy way of rigging the draw, even though that is completely untrue.

The side effect of asking citizens to question the authority of experts and politicians is that they also question the designers of deliberative events! Mind you, in some cases (eg. Gordon Brown's "citizens' juries") that would be a good thing.

You raise the question of attrition after selection. It's a challenge for accurate stratification, especially for last minute drop-outs. Go to this page to find the articles about how we did it at the CP: http://bit.ly/egYLP

I'm an IAP2 member. Feel free to call me anytime.

thanks for getting in touch. And thanks for making such a thoughtful reply to my post. I'll start with an admission, I've noticed that unless posts are a little provocative they tend to be ignored in much the same way as our community forums are ignored when the issues are benign. And there is nothing worse than being ignored! The post was framed as a beginning rather than an ending of what I hoped would be a conversation about this issue.

From reading your reply I think we are pretty much in furious agreement about just about everything. I've always been a big fan of Citizen's Jury processes precisely because of their ability to overcome the what can be built up as a false dichotomy between "open-access" and "by-invitation" processes. The deliberative nature of this kind of process is critical to the integrity of the policy recommendations. My major concern is more with the use of SRS in non-deliberative polling where there is no opportunity for learning. There seems too high a risk of "push-polling" being wrapped up in the guise of "good science".

Your comment that we can't ask people questions about their values to help inform the sample without upsetting them has my curiosity piqued. Has there been much research around this issue? Australian's aren't too keen to discuss religion and closely held personal beliefs so I imagine that this must be a something of a conundrum.

An issue I didn't raise in the post is the question of how random a random sample can ever be given the ability of people to "opt-out" of the process at the outset. Has there been much research around the potential for bias in the sample as a result of the willingness or otherwise of individuals to even begin a conversation with the researcher?

I note your comment regarding the promotion of online discussion forums. This is something we confront everyday as part of our business. It can be hard, particularly when the issue being discussed doesn't have an obvious and immediate relevance to an individual. We find it much easier to attract attention to forums about infrastructure projects for example, than Annual Management Plans – which potentially have a much greater impact. We're experimenting with social media and tend to recommend mainstream media as the main driver of traffic.

BTW. I'm in Melbourne – Brunswick East. Would love to catch up personally for a metaphorical coffee to chat about this and the Citizens Parliament. I'm the local IAP2 contact and think our membership would be very interested in some sort of forum.

You've written a provocative post and I agree that it is worthwhile to explore the merits of SRS. You may know that I wrote the software to do the SRS for the Citizens' Parliament. We stratified on gender, age, education, locality (1 per federal HoR electorate) and aboriginality.

My interest is in public deliberative processes that generate policy recommendations that are acceptable by the whole community, bureaucracy and elected government. Of course, there is a place and a need for stakeholder processes for which SRS would be silly. But I'd argue that some difficult issues are better addressed with the former than the latter. A process like a Citizens' Jury overcomes the duality by bringing stakeholder interests in to persuade the minipublic.

Confidence in public deliberation processes is a social construction, and one way to build that confidence is through SRS. Of course, it would be technically better with conscription and belief-survey (as you suggest), but we can't have those without upsetting people and defeating our purpose! So we go with what we can do and benefit by improved deliberative potential and in public perception of the process.

I agree that in absolute terms age and gender mean little. But if we can't reliably survey belief systems, then we can use age and gender as second-order selectors, because beliefs tend to cluster with them. So stratification helps spread the diversity.

Yes, SRS tends to reduce the influx of "usual suspects". Certainly the forum would benefit by their hard questions. But invariably they come with strongly-held prescriptions and an unwillingness to behave civilly in a deliberative forum.

There is an additional thing that we ask of deliberating minipublics, especially in a small jury: we ask them to look beyond just their private views but to the diversity of views in the community that will be impacted by their recommendations. If they are randomly-selected, they identify themselves as representatives, and are more inclined to be generous and exploratory in their consideration of different perspectives.

I agree with you that the usual suspects are well-meaning and should be able to make a contribution. The question is how? The solution we tried at the Citizens' Parliament was to have a public-access online discussion forum in parallel and informing the deliberating minipublic. (It needed promotion, though…)

–Ron.