SURVEYS AREN’T SCIENCE

A recent mention of the ‘survey’ method of ‘research’ is worth a brief look here.

The ‘New York Times’ published an article * a few days ago, on Page One.

The piece talks about noted psychiatrist Robert Spitzer, prime mover behind the now-indispensable (if highly-debated) ‘Diagnostic and Statistical Manual’ (“DSM”) now in its 4^th-Revised Edition and soon to be replaced by a 5^th.

The particular point of the piece is to discuss Spitzer’s now-public and formal apology for embracing – some years ago – ‘reparative therapy’ for gay persons.

It is not my purpose here to get into the complexities of ‘reparative therapy’.

But as so often happens in conceptual matters, an amazing bit was revealed almost inadvertently, that bears far more widely and yet also directly on matters of SO concern (and should also be of concern to the entire Citizenry, to the public).

Spitzer had, some years ago, gotten somehow involved with some pro-‘reparative therapy’ professionals and decided to join in.

He did some ‘research’.

He did this ‘research’ by conducting a ‘study’.

He conducted this ‘study’ by doing a ‘survey’.

His research Question was: Did this ‘reparative therapy’ approach really work on people?

And he ‘surveyed’ them. That is to say (p.A3) he collected a list of 200 people who had had the therapy (comprised of folks who had gone through this therapy and were in the database of the pro-therapy organizations). He then called each of them on the phone. And he asked them a list of questions he had devised, about their sexual urges and thoughts and experiences before and after they had participated in the therapy (the article says he “interviewed them in depth”).

And they answered his questions.

Spitzer then took his collection of answers and compared the before and after descriptions he had been given. “The majority” of them “gave reports” indicating that they had gone from a predominantly or exclusively homosexual orientation to a predominantly or exclusively heterosexual orientation in the past year.

On the basis of this ‘study’, he delivered a paper to a major psychiatric conference in 2001, reflecting his conclusion that ‘reparative therapy’ worked.

Yes, there was a political uproar from the gay community immediately. But that’s not my point or subject here.

Spitzer was quickly taken to task by scientific and professional colleagues for the grossly flawed methodology of his ‘study’; many of the criticisms were “merciless”. People had been asked about their memory / of feelings / that they had had years before. Some of the people were ‘activists’ (who might presumably be perhaps motivated to give answers that served the purposes of their cause). Some had undergone professionally-provided therapy but many had simply undergone ‘counseling’ or discussion with some paraprofessional provider or had simply done some “independent Bible study”.

There were many non-professionals who objected simply because of how his study might be ‘interpreted’ by politically unfriendly (in this case ‘conservative’) elements; in other words, they merely objected to the possible consequences of this particular study, especially in light of their political agendas.

But the key problem – as many of the professional researchers pointed out ‘mercilessly’ – was the integrity of the study itself: you can’t simply ask people questions over the phone – especially if they can assume that their answers won’t ever be checked or independently evaluated and corroborated – and expect thereby to get ‘facts’. At best, you get opinions or vague memories or – far more ominously – ‘motivated’ responses specifically tailored by the respondent to support some ulterior motive or agenda.

(And all of this presumes that you as the ‘researcher’ haven’t already figured all that out, and specifically construct your questions or your call-list or both precisely to tap into all of this in order to make sure you get the biggest ‘numbers’ you possibly can.)

But in any case, you can’t consider your ‘results’ to be scientifically definitive or reliable on their face; you have to then go and corroborate independently each and all of the responses. Otherwise, you have nothing more than an ‘opinion survey’ and what you most certainly do not have is a scientifically credibly and legitimately characterized ‘study’ or ‘research’ or ‘scholarship’.

As the article puts it: “Simply asking people whether they have changed is no evidence at all of real change”.  And then, the article continues: “People lie, to themselves and others. They continually change their stories, to suit their needs and moods.” To say nothing of any more specific ulterior motives and objectives and agendas they might quietly have embraced.

Just so.

You can’t simply take respondents’ word for it when you ask them your questions.

But now, moving on from this article, I point out that such ‘survey science’ is precisely what has fuelled so many elements of the ‘facts’ that drive the SO Mania Regime.

How much of the ‘scholarship’ and ‘science’ that advocates have pushed toward the media and the legislators has been the result of nothing more than ‘surveys’? Surveys whose ‘answers’ not only create ‘numbers’ but also – if the survey-derived ‘numbers’ are far greater than the actual numbers of reported cases – create the ‘justification’ for claims that for every single reported case there are 10 (or 100 or 1000 or 10,000) ‘unreported’ cases.

Further – and this also demonstrates a certain political bias in the media and among legislators – what happens when you apply this to a phenomenon such as ‘sex offenses’ or ‘rape ’or ‘sexual abuse’ (however your questionnaire might define those elastic terms)? Suddenly there are untold hundreds of thousands or millions of ‘unreported’ cases.

And does not the same reality hold for respondents of sex-abuse surveys as holds true for the respondents of the reparative-therapy survey: you can’t trust the answers you get because you can’t simply ‘trust’ and ‘believe’ the persons making those answers … ?

But, of course, in the SO Mania Regime, Correct victimist dogma insists that you precisely can’t question the answers you get – and so you can neatly accept as gospel truth the astronomical ‘numbers’ you wind up with.

So scientific integrity is all well and good if you are going to be stopping something you don’t want to see (e.g., the acceptance of ‘reparative therapy’, which is anathema to a politically powerful advocacy). But scientific integrity is absolutely an obstruction and some form of evil collusion if it gets in the way of ‘believing victims’, which is the primary goal of a politically powerful advocacy or – more accurately – a combination of assorted allied advocacies and lobbies.

This is a gross and reprehensible double-standard that has derailed and deranged the integrity of scholarship and research and ‘science’, of legislative and jurisprudential praxis, of media assessment and coverage, and – worst of all by far – the integrity of the public’s competence to make informed judgments based on accurate information.

‘Surveys’ are not scientific nor are they science nor scholarship nor research. Uncorroborated, they are nothing more than a focused form of hear-say.

I have seen it asserted that sometimes you simply can’t do the field-research and so you “have to” rely on surveys – as if somehow the fact that you can’t (or won’t, or don’t want to) do scientific work means that whatever you do manage to do is thereby legitimately characterizable as ‘science’ and ‘research’ and ‘scholarship’ and your results are ‘facts’. As if ‘surveys’ can become justifiable as science ‘by default’: I can’t or don’t want to risk doing actual research and evaluation, but I think I can get what I want with a ‘survey’ and so the survey I conduct is ‘science’.

Imagine a fire department that somehow runs out of water at a fire, commandeers a nearby gas tanker truck, and starts pumping gasoline on the fire: yes, all the usual actions are being taken (firemen spraying hoses, fire engines pumping out streams of liquid into hoses) but in reality you most certainly are only mimicking fire-fighting operations and you most certainly are not putting out the fire. Just the opposite, in fact.

But this is a scam that has served the purposes of many ulterior interests and continues to do so.

And thus, the SO Mania continues its curious life as a fire that just doesn’t seem to go out – and indeed seems to only get worse, no matter how much money and ‘science’ the government sprays onto it.

NOTES

*The story is entitled ‘Psychiatry Giant Sorry for Backing Gay ‘Cure’’; in the print edition it appeared on Saturday, May 19, 2012, on page A1 and continued on page A3.

ADDENDUM

As an example of research skewed for the purpose of reaching a particular desired conclusion (and no other), you can examine this new 14-page formal critique of a recent Army effort to wish away the lethal difficulties of stress among troops.