According to the Office of Research Integrity, US Department of Health & Human Services:

However, consistent with the basic tenets of ethical writing and scientific objectivity, we have a responsibility to cite all relevant material, even work that may contradict our own position. Failure to do so compromises our professional obligation to remain unbiased and is antithetical to the primary mission of a scientist’s search for truth.

• Unacceptable construct validity study
• Poor concurrent validity study

MISLEADING STATEMENTS

There are no peer-reviewed validity publications on any of the many CANS that relate to construct or concurrent validity. Yet, Lyons claims “CANS has demonstrated strong reliability and validity” (Kisiel, Fehrenbach, Small & Lyons, 2009, p. 147).

• He cites to an edited compilation (Lyons & Weiner, 2009) in Kisiel, et al. (2009).
• In Obeid & Lyons (2011) he cites another book stating: “Substantial reliability and validity information now exists on the CANS (Lyons, 2009)” (p. 74).

He often sites to his many books (not peer-reviewed) for this primary source evidence. For example:

• But, his 2009 book only cites a number of conference presentations (not peer-reviewed) and his 2004 book.
• So we follow the paper trail to the 2004 book. The entire “validity” section from the 2004 book is copied to the right. He cites a parent’s presentation (Julie Hdalio – see her description in Lyons, 2009, p. 101) for independently assessed CANS concurrent validity with CAFAS. Really?

KNOWN LIMITATIONS

Lyons has conducted many important validation analyses on CANS, presented several at conferences and failed to acknowledge the concerns in any publication.

• Miller, Leon & Lyons (2007): shows an admitted failure to document construct validity, that a year later, Lyons said should be done. “future research should establish the … construct validity of the CANS scales used here through exploratory and confirmatory factor analysis” (Sieracki, Leon, Miller, & Lyons, 2008, p. 806). CANS dimensions like Strengths are promoted today, but knowingly flawed.
  • Read it yourself.
  • A second, larger dataset was similarly analyzed, but not reported (cf. Lyons, McClelland & Jordan, 2010, p. 726).
  • Conclusion: Failed Construct Validity

• Dilley, Weiner, Lyons & Martinovich (2007): shows that only one item on CANS shares at least half the common variance with a CAFAS score (Substance Abuse) which no longer exists on CANS (now Substance Use). Why are all the other CANS items not listed? Count them here.
  • Conclusion: Failed Concurrent Validity

False Citations

One unethical way to make up scientific data or knowledge is to claim something, and then cite a reference that has nothing to do with the topic, or cite to a reference that actually suggests the opposite. CANS’ Publications are replete with this type of Research Misconduct. In total we count more than forty-nine (49) false citations. For example, in the CANS’ Bibliography itself, twenty (20) of the fifty-two (52) publications listed have nothing to do with CANS and are actually research on a different topic. For example, Lyons & Rogers (2004) [23] makes no reference to CANS. It is a discussion of how the child welfare system has become the primary system of care for children with behavioral health issues. Twenty-seven (27) of the false citations are used as proof of CANS’ validity.

For example, in 2009 [33], Lyons wrote: “Two published audit studies have demonstrated its high field reliability (Anderson, Lyons, Giles, Price & Estes, 2002[03]; Lyons, Rawal, Yeh, Leon, & Tracy, 2002[R])” ([33], p. 74). [R] is a study using a tool called the CSPI, not CANS. And while Lyons repeatedly tries to conflate CSPI research for CANS research they are different tools, and in some studies, they are both used (cf. [09] and [25] where the CSPI was used for study eligibility screening, and CANS was used for service planning). The CSPI has fewer items and a different rating scale.

In another example, Epstein and colleagues in 2015 cited six (6) non-CANS studies as support for the validity of CANS.  Two of which, he was an author [08]. In another Epstein publication he further cited eight (8) non-CANS studies for the validity of CANS [07].

When CANS’ researchers cite actual CANS articles for the validity of CANS, they cite non-peer-reviewed articles that are not primary sources of validity data. For example, in 2010, Lyons and colleagues wrote: “In terms of validity, the CANS’ dimension scores have been shown to correlate with other measures of child status, such as the CAFAS and Child Behavior Checklist (Lyons 2004[26]) and has demonstrated relationships to service use and level of care decisions (e.g., Lyons 2009[34])” ([35] p. 721).

Tracking down these references, Lyons’ 2009 book [34] is not a primary source of research data on any subject. Ethics and ORI guidelines require that researchers cite primary sources, which CANS researchers fail to do. Interested readers are forced to follow rabbit holes in a search for the primary sources which do not exist. [34], for example, has only two small paragraphs allegedly reviewing CANS APA-defined validity terms. In full it says:

VALIDITY: Several types of validity have been studied and established for the CANS. Face validity is demonstrated by its widespread acceptance in a large variety of child-serving systems. There has been remarkably little resistance from family advocates and clinicians for most implementations. The approach clearly makes sense to those working directly with the children and families. Most resistance to CANS implementations actually come from individuals schooled in traditional psychometric measurement approaches who are uncomfortable with the communimetric approach.

In terms of construct validity, CANS dimension scores have been shown to correlate with other measures of child status, such as the CAFAS and the Child Behavioral Checklist (Lyons, 2004). These correlations are highest when the context is the same for all children in the sample. When some children in a residential placement and others are in the community, the correlation between the CANS and CAFAS is much lower, as the CANS does not report setting effects as meeting the needs of children, whereas the CAFAS does (Lyons, 2004). In other words, a child who is going to school at a campus-based residential program because staff wake him or her up and ensure that he or she attends the on-campus school would be seen as fine on the CAFAS, but on the CANS could still be seen as having school attendance needs.” ([34] p. 103)

These are fabricated conclusions. There is no research to support the face validity, lack of resistance, or conclusion that those that resist want to use a well-grounded psychometric tool. Lyons (2004 [26]) does not present any data related to the CBCL or Child Behavioral Checklist, but does present alleged data regarding the CAFAS correlations (reviewed above). The low correlations are explained away as a result of CANS evaluating the child in the hypothetical environment without supports. CANS “defines a person’s functioning without interventions in place” (Lyons & Israel, 2017 [X], p. 1). Guessing how a child might behave if supports are removed is likely to introduce substantial error into the measurement process.

Buried Validity Data

Multiple, failed attempts to validate CANS have been buried. This violates APA ethics as well as ORI Guideline 21.

Miller, Lyons & Leon (2009 [V]) present, in a brief conference presentation, the results of a construct validity test of an undefined version of CANS. From the degrees of freedom listed, it is likely the 41-item version used in some locations in the early 2000s (cf. Anderson et al., 2003 [03]). The analysis concluded that no acceptable factor structure of CANS could be created, and the dimensions promoted by Lyons (like Strengths and Risk) are not valid. The study also concluded that a central tenant of Lyons’ idiosyncratic Communimetric theory (Lyons, 2009) is not supported (i.e., weighting items equally so that they can be used individually). The authors, including Lyons, explicitly recommend that individual items not be used. Lyons has ignored his own conclusions and continues to promote unacceptable factor structure and individual item uses of CANS data. The LOC algorithm is just one such example that has major clinical implications.

This failed validity study has not been cited in any CANS’ Publications. However, the same authors, a year later, in 2008 [47], write that this very study should be conducted: “future research should establish the … construct validity of the CANS scales used here through exploratory [EFA] and confirmatory factor analysis [CFA]” (Sieracki, Leon, Miller, & Lyons, 2008 [47], p. 806). The study, however, had been conducted. Looking past their own, failed results, the authors promote CANS as valid:

The CANS has been documented to be a reliable and valid measure (Lyons, Rawal, Yeh, & Tracy, 2002[R]). It is a useful tool for predicting the level of care that a child is placed in and is correlated with a similar measure of child outcomes (the Child and Adolescent Functional Assessment Scale (CAFAS); Hodges, McKnew, Cytryn, Stern, & Klein, 1982). It has also demonstrated to be sensitive to change, and therefore, is considered a useful outcome-monitoring tool. (p. 802)

Note that some two decades before the development of CANS, the authors cite Hodges’ work, on the validation of her CAFAS tool, for the proposition that CANS has predictive validity.

Two other factor analytic analysis of CANS have been inappropriately reported. Kisiel and colleagues attempted one in 2009 [I], but they do not present any essential statistics like eigen values or factor loadings. Their study results, even if presented with the relevant EFA statistics, without a CFA confirmation study, is unlikely to be valid. Costello and Osborne (2005) tested the effect of sample size on the results of factor analysis. Their results suggest that Kisiel’s meager sample of just over 100 children to evaluate almost as many CANS items would have less than a 10% chance of being accurate.

A second, larger, and maybe more useful dataset was also analyzed, but not reported (cf. Lyons, McClelland & Jordan, 2010 [35], p. 726). Those authors should be encouraged to share the raw data. In addition, since virtually every state in the country has a large clinical dataset, these could be explored with EFA models and confirmed in multiple large CFA studies. The results are unlikely to yield acceptable results.

Lack of Peer Review

Lyons claimed that the publications listed in the CANS’ Bibliography is an exhaustive list of the independent, peer-reviewed, articles on CANS. Only nine (9) are peer reviewed.

Twenty-two (22) are not related to CANS’ research. An additional three (3) are unpublished dissertations. Nine (9) are not primary source research, like books and book chapters. Five (5) focus on using CANS in its original format as a retrospective chart review tool and not relevant to Family First. Three (3) additional articles were in Lyons’ edited journal and do not appear to have been independently peer-reviewed. One (1) was related to an extremely different version of CANS focusing on childhood trauma. This leaves nine (9) apparently peer-reviewed publications.

All nine of these publications squeaked by the peer review process by either ignoring the lack of validity of CANS or by making false claims of validity. Each of the below-referenced citations have been reviewed and do not stand for the proposition they claim to support.

Anderson [03] claimed:

Literature supports the reliability and the validity of the Child and Adolescent Needs and Strengths-Mental Health (CANS-MH) as a chart review methodology across a number of settings and documents adequate interrater reliability among researchers (Leon, Lyons, & Uziel-Miller, 2000 [J]; Leon, Lyons, Uziel-Miller, & Tracy, 1999 [L]; Lyons, Howard, O’Mahoney, & Lish, 1997 [O]; Lyons, Mintzer, Kisiel, & Shallcross, 1998 [Q]; Lyons, Uziel-Miller, Reyes & Sokol, 2000 [21]). (p. 281)

Effland [05] claimed:

The CANS has been found to be reliable and valid (Lyons 2009[34]). (p. 738)

Epstein [07] claimed:

Previous research demonstrates this measure to have adequate inter-rater and internal consistency reliability (Anderson et al., 2003[03]; Epstein et al., 2009[06]; Leon et al., 2000 [J]; Leon et al., 1999 [L]; Lyons et al., 2002; Lyons et al., 2004[27]), and concurrent, discriminate, and predictive validity (Epstein et al., 2009[06]; Leon et al., 1999 [L]; Lyons et al., 2001 [N]; Lyons et al., 1997 [P]; Lyons et al., 2000a[S],b[21]; Lyons et al., 2004[27]; Park et al., 2009[40]). (p. 252)

Kisiel [17] claimed:

A substantial body of research exists on the measurement properties of the CANS (Lyons, 2004[26]). Anderson, Lyons, Giles, Price, and Estes (2002)[03] indicated that the CANS is reliable at the item level so individual items can be used alone in data analyses. Further, the CANS has demonstrated strong reliability and validity in field applications in child welfare, mental health, and juvenile justice (Leon et al., 2008[K]; Lyons et al, 2000[21]; Lyons & Weiner, 2009[31]; Sieracki, Leon, Miller, & Lyons, 2008[47]). Initial psychometric studies suggest that the domains of the IDCFS CANS comprehensive exhibit strong reliability and validity (Kisiel, Blaustein, Fogler, Ellis, & Saxe, 2009[I]; Lyons & Weiner, 2009[31]).

Lyons [35] claimed:

The CANS has well established reliability and validity (e.g., Anderson et al. 2002[03]; Lyons 2004[26]). … In terms of validity, the CANS has a documented associated with similar measures of functional status such as the CAFAS and Child Behavior Checklist (Lyons 2004[26]) and has demonstrated relationships to service use and level of care decisions (e.g., Lyons 2009[34]). (p. 721)

Sieracki [47] claimed:

The CANS has been documented to be a reliable and valid measure (Lyons, Rawal, Yeh, & Tracy, 2002[R]). It is a useful tool for predicting the level of care that a child is placed in and is correlated with a similar measure of child outcomes (the Child and Adolescent Functional Assessment Scale (CAFAS); Hodges, McKnew, Cytryn, Stern, & Klein, 1982). It has also demonstrated to be sensitive to change, and therefore, is considered a useful outcome-monitoring tool. (p. 802)

Weiner [50] claimed:

high inter-rater reliability when completed by trained professionals through chart review or through direct contact with the child being assessed (.81; Anderson, 2003 [03]) (p. 1200)

Lyons [25] and Weiner [51] decided to ignore the lack of CANS validity and made no comments.

All nine of these studies are convince-sample research studies that take available state data during a window of time. None provide any controls, like risk-adjustment of the data to control for non-random effects that these convenience samples introduce. None discuss the limitations and threats to external validity.

Of these nine (9) peer-reviewed articles, Lyons is an author on six (6)  ([03], [16], [25], [35], [47], [50]). Those that worked for or with Lyons at Chapin Hall are authors on two (2) additional ([07], [51]). This leaves potentially one (1) study that was peer-reviewed and independent ([05]).

In this lone, CANS study, Effland, Walton & McIntyre’s (2011 [05]) reported: “The purpose of our study was to examine the relationships among the implementation of necessary support conditions for wraparound, wraparound fidelity and youth outcomes.” (p. 737) Like the state of Indiana that required CANS be used throughout the state, the authors trusted Lyons that CANS was valid and cited his book: “The CANS has been found to be reliable and valid (Lyons 2009[34])” (p. 738). It is not known what version of CANS was used in this study.

Effland did not appear to conclude that CANS was very useful for the study’s objectives. The authors conclude that wraparound fidelity scores were the strongest predictor of a change in client needs, rather than CANS baseline assessments, supporting Sieracki’s [47] conclusion that CANS’ baseline scores predict a very small percent of variance (<6%) in outcome scores.

In fact, the external validity of each of these CANS’ studies are poor.

The American Psychological Association defines external validity as:

the extent to which the results of research or testing can be generalized beyond the sample that generated them. For example, if research has been conducted only with male participants, it cannot be assumed that similar results will apply to female participants. The more specialized the sample, the less likely will it be that the results are highly generalizable to other individuals, situations, and time periods (dictionary.APA.org).

We can find no CANS’ Publication that has heeded the well-grounded external-validity standards of science. Two studies discuss the external validity problem, but then ignore the limitation. Discussing the limitations of her study, Cordell [D] states, “the present results apply elsewhere only insofar as the populations served and CANS items used match the populations and the 129 CANS items investigated here” (Cordell, Snowden & Hosier, 2016 [D], p. 133). However, in choosing CANS for her study and citing its psychometrics in the relevant “measures” text of the manuscript, Cordell states: “Published research provides evidence that the CANS is sound psychometrically (Hodges, Kline, Stern, Cytryn, & McKnew, 1982; Lyons, Rawal, Yeh, Leon, & Tracy, 2002 [R])” (p. 129). Neither citation has anything to do with CANS, let alone the version she used.

Dunleavy & Leon (2011 [E]) noted that “single informant strategies are associated with threats to external validity” (p. 2353), but this concern did not alter their study design.