M E M O R A N D U M October 2, 2012 To: Members of the Charter Schools Committee From: Joseph W. Belluck, Chair, Charter Schools Committee Subject: Adoption of Methodology to Set Enrollment and Retention Targets Pursuant to Article 56 of the Education Law Action Requested The proposed resolution authorizes the adoption of a methodology to set enrollment and retention targets for charter schools for students with disabilities, English language learners, and students who are eligible applicants for the federal free and reduced price lunch program in order to implement Education Law subdivisions 2851(4)(e), 2852(9-a)(b)(i) and 2855(1)(e). Resolution I recommend that the Charter Schools Committee adopt the following resolution: Whereas the SUNY Charter Schools Institute (the “Institute”) working with the New York State Education Department drafted a proposed methodology to set enrollment and retention targets for charter schools for students with disabilities, English language learners, and students who are eligible applicants for the federal free and reduced price lunch program consistent with the New York Charter Schools Act of 1998 (as amended, the “Act”) including Education Law subdivisions 2851(4)(e) and 2852(9-a)(b)(i), which methodology is described in the Charter School Enrollment and Retention Targets Technical Report dated June 28, 2012 (copies on file in the Office of the Secretary of the University and the Albany office of the Institute) (the “Proposed Methodology”), and presented same to the Charter Schools Committee (the “Committee”); now, therefore, be it Resolved that the Committee finds the Proposed Methodology is sufficient and meets the requirements of the Act; and be it further Resolved that the Proposed Methodology, be, and hereby is, adopted by the Board of Trustees to develop enrollment and retention targets within the meaning of the Act; and be it further Resolved that the Institute be, and hereby is, directed to prescribe enrollment and retention targets within the meaning of the Act prior to the submission of applications for: 1) new charters to be issued pursuant to Education Law subdivision 2852(9-a); and 2) charter renewal. Background Pursuant to Resolution No. 2012-038 dated June 12, 2012, the Board of Trustees delegated authority to promulgate and revise policies related to the authorizing of charter schools to the Charter Schools Committee (the “Committee”). Amendments to New York Charter Schools Act of 1998 (as amended, the “Act”) effective May 2010 required the Board of Trustees and the Board of Regents to develop charter school enrollment and retention targets for the categories of students listed in the resolution for use in conjunction with approving new charter school applications and renewal applications as follows: When developing such targets, the board of regents and the board of trustees of the state university of New York shall ensure (1) that such enrollment targets are comparable to the enrollment figures of such categories of students attending the public schools within the school district, or in a city school district in a city having a population of one million or more inhabitants, the community school district, in which the charter school is located; and (2) that such retention targets are comparable to the rate of retention of such categories of students attending the public schools within the school district, or in a city school district in a city having a population of one million or more inhabitants, the community school district, in which the proposed charter school would be located. Education Law § 2851(4)(e) (describing renewal application requirements). The proposed methodology adopted by this resolution will provide a means to establish targets. The actual targets will be set by the Institute using the proposed methodology, district school data and the grade configuration and enrollment numbers of individual SUNY authorized charter schools. Education Law subdivision 2855(1)(e) states that the Board of Trustees may revoke the charter of any school that repeatedly fails to meet the targets and which does not meet the requirements of certain statutory defenses. Charter School Enrollment and Retention Targets Technical Report Updated June 28, 2012 New York State Education Department Charter School Office and the SUNY Charter Schools Institute This report describes the current proposed methodology behind the calculation of charter school enrollment and retention targets for free- and reduced-price lunch eligible students, limited English proficient students, and students with disabilities as required by New York Education Law Section 2851(4)(e).1 According to the stipulations of this law, enrollment and retention targets are calculated for students in all districts at the level of the school district, and in the case of New York City, the Community School District (CSD). The specific rules applied to setting the targets are discussed in detail below. This report also discusses a proposed way to apply the targets to charter schools. Calculating Enrollment Targets Enrollment targets are based on student demographic and school enrollment information from New York State Education Department student-level data files. Students who enrolled in a traditional district or charter school in the 2010-11 school year were included in the target setting sample. Students explicitly omitted from the target sample include those enrolled full-time in: a) Boards of Cooperative Educational Services (BOCES) schools, b) private schools, including those for students with disabilities or other special education needs, c) District 75 in New York City, and d) PS 42 and PS 84 in Buffalo Public Schools. Because the timing of student classification differs across schools and districts, and to ensure that schools are not “penalized” for declassifying students over time, students were identified as free- or reduced-price lunch students, limited English proficient students, or students with disabilities if they were ever classified as such in the NYSED student-level data files in school years 2009-10, 2010-11, or 2011-12. This approach, by definition, gives credit to schools and districts that enrolled classified students at any point during the most recent three school years. After restricting the target sample to the population of interest, students were assigned to their school and district of attendance on October 1, 2010. Sample targets were also tested using observed enrollments on April 1, 2011 and June 30, 2011, and on these dates in other school years. These analyses validated the use of October 1st enrollments for setting targets since alternative enrollment snapshots yielded comparable targets. Enrollment targets were created through the following two-step process: 1. The total number of students from each target classification group in each potential grade span was aggregated at the district level. 2. The number of classified students (obtained in step 1 above) was then divided by the total number of students enrolled in the district and grade span, yielding a unique proportion of classified students for each district and grade span configuration. Calculating Retention Targets The process for establishing retention targets involved starting with the same set of student-level data files as for setting enrollment targets. Calculating these targets also employed the same set of initial sample restrictions, i.e. restricting the target sample to include only students who enrolled in traditional district or charter schools, and students were also identified as free- or reduced-price lunch students, limited English proficient students, or students with disabilities if they were ever classified as such in the NYSED student-level data files in school years 2009-10, 2010-11, or 2011-12. After restricting the target sample to the population of interest, retention targets were created through the following four-step process: 1. The total number of students from each target classification group in each potential grade span that experienced a Type 2 discharge between October 1st of a given school year and September 30th of the subsequent school year was aggregated at the district level.2 2 2. The number of classified students with Type 2 discharges (obtained in step 1 above) was then divided by the total number of classified students enrolled in the district and grade span between October 1st and June 30th of the given school year, yielding a unique withdrawal rate for classified students for every district and grade span combination. 3. The withdrawal rate was then subtracted from 100%, corresponding to the total proportion of students initially enrolled, to obtain a unique retention rate for classified students for every district and grade span combination. Retention rates were separately calculated for each district from 2009-10 to 2010- 11, and from 2010-11 to 2011-12. 4. A 2-year average retention rate for each district was calculated by averaging the rates from 2009-10 to 2010-11, and from 2010-11 to 2011-12. With the exception of Albany City School District, Hempstead Union Free School District, and Middletown City School District, this 2-year average retention rate was used in place of a single year rate because enough variation was observed in school-level rates across years to justify an averaging approach to achieve more precise retention targets. Retention targets for Albany City School District and Hempstead Union Free School District were based solely on retention from 2009-10 to 2010-11 because of data quality concerns in the most recent school year that resulted in systematically lower retention rates in the district from 2010-11 to 2011-12. Conversely, retention targets for Middletown City School District were based exclusively on retention from 2010-11 to 2011-12 because of data quality concerns in the earlier period that resulted in systematically lower retention rates in the district from 2009-10 to 2010-11. Calculating Standard Errors and Lower Limits (“Effective Targets”) Fluctuations in student populations and sampling frames are likely to yield natural variation in school-level enrollment and retention rates in any given year. A lower limit, or “effective target,” calculated for each target, accounts for this expected variation and thus reflects the lowest possible enrollment or retention rate a school must report to meet its target. Effective targets are set as a one-sided 95% confidence interval around the target.3 A one-sided interval is used instead of a more traditional two-sided interval because the question of interest is whether schools meet or exceed the target, not whether schools fall within the upper or lower bound where ET is the effective target, T is the unadjusted target, n is the total number of students (for enrollment targets n is the schools size, while for retention targets n represents the number of classified students enrolled), and 1.645 corresponds to the z-score for a standard normal probability distribution in which the likelihood of the target falling below the effective target is less than or equal to 5%. Summary of, and Response to, Public Comment Received Proposed Methodology to Establish Enrollment and Retention Targets for At-Risk Students in New York State Public Charter Schools Updated July 12, 2012 Categories assigned to comments below are for ease of reference for readers Three-year Look-back for Setting Targets vs. Data on Meeting Targets Comment 1: A number of comments addressed the proposed “three-year look-back” included in the proposed methodology, stating: a) three years is not enough time to give charter schools “credit” for working to declassify students as quickly as possible; b) three years is not enough time to give charter schools credit for quality response to intervention programs that enable them to meet student needs without referral to the district Committee on Special Education (CSE); c) according to the technical paper, schools are really only given “credit” for the current year and two prior years and therefore the name is misleading; and d) the three year look-back, as proposed, creates a perverse incentive for charter schools to potentially over-classify students to meet proposed enrollment and retention targets. Alternate recommendations for the final methodology suggested by commenters included: tracking students’ progress from the time they enroll; tracking students identified for Tier 1 or Tier II support who were not ultimately referred by the CSE; and tracking declassifications at both charter schools and district schools and rewarding charter schools for exit rates that exceed those of the district. Discussion: The “three-year look-back” in the proposed methodology is designed to ensure data quality in setting the targets. In constructing the proposed methodology, SUNY and SED staff (“the working group”) sought to create time bound data terms that were statistically valid. The comments provided address the time-period authorizers should consider when comparing school data toward meeting the targets. The comments raise important school-based practice realities related to instructional methodology, preferred pedagogical approaches to interventions and supports, and theories of action related to school design. These issues are all local control issues linked to school-based autonomies that public charter schools enjoy under Education Law and the contracts that they hold with their authorizers. The determination of whether or not a school has met a particular target will be addressed in greater detail in the accountability policies and linked practices to be developed by each authorizer, including data and evidence supporting “best faith efforts” to meet targets Changes to the proposed methodology: None. This comment will be considered during related accountability policy and practice development and implementation. City-wide vs. CSD Data Comment 2: Several comments stated that the rationale for using city-wide data as opposed to community school district data to set targets for students with disabilities in New York City is flawed and does not comply with the requirements of Education Law. One comment notes that in September 2012 the New York City Department of Education will institute a city-wide policy to ensure that most students with disabilities are served by their zoned school in the CSD where they reside. Another comment notes that students designated to receive Special Education Teacher Support Services (SETSS) do not participate in a city-wide placement process and that only in a rare cases are students designated to be in integrated co-teaching (ICT) classrooms placed outside of their community school district. Comments suggest that there is no reason that the CSD should be the unit of comparison. Discussion: The statute indicates that in school districts that serve more than one million students, authorizers should compare the enrollment and retention proportions of charters to the community school district (CSD) rather than to the entire school district. However, historically the New York State Education Department has not made disaggregated data on special education by CSD available. Additionally, there are concerns that there are disparities in the numbers of students with disabilities assigned to particular schools for a broad variety of reasons, which is particularly problematic for high schools given the city-wide admissions process in New York City. Changes to the Methodology: After consideration of public comment, review of the statutory language, and available data, targets in New York City will be set by CSD rather than city-wide for students with disabilities. Comment 3: Several comments stated that using city-wide data as opposed to community school district (CSD) data to set targets for high school grades does not comply with the requirements of the law. Discussion: In New York City, high school admission is carried out by a city-wide choice process, and therefore targets set across the entire city in grades 9-12 may best align with the high school admissions process. However, the statutory language clearly indicates that targets are to be set by CSD. For this reason, all targets, even in NYC in high school grades, are set at the level of the CSD. Changes to the Methodology: After consideration of public comment and a review of the statutory language, targets in New York City will be set by CSD rather than city-wide for students with disabilities. Special Districts Comment 4: One comment notes that the technical memo indicates that District 75 in New York City was excluded from the analysis. The comment argues that PS 84 Health Care Center in Buffalo should be treated the same way. Discussion: The New York City Department of Education’s publicly available definition of District 75 indicated that it “provides city-wide educational, vocational, and behavior support programs for students who are on the autism spectrum, have significant cognitive delays, are severely emotionally challenged, sensory impaired and/or multiply disabled.” District 75 consists of 56 school organizations, home and hospital instruction and vision and hearing services located at more than 310 sites in the Bronx, Brooklyn, Manhattan, Queens, Staten Island and Syosset, New York. The PS 84 Health Care Center in Buffalo, while considerably smaller than District 75, is designed to provide an alternative, individualized, learning environment for SWD’s to develop and maximize their abilities to become more independent, contributing members of society. As such, for purposes of target setting, District 75 and PS 84 should be treated similarly. Additionally, PS 42 Occupational Training Center in Buffalo was also identified as a similar setting which should be excluded from the analysis. Changes to the Methodology: PS84 and PS42 in Buffalo, identified as analogous settings to District 75 in New York City, will be excluded from the set of schools contributing to the setting of enrollment and retention targets. Accounting for School-Level Variability Comment 5: A comment stated that the proposed methodology uses aggregate district enrollment rather than school level enrollment and thus does not account for school-level variability within a district. Discussion: The working group notes that the proposed methodology does account for variability across the schools in a particular district by giving more weight to larger schools in the district and less weight to smaller ones, thereby more accurately approximating the aggregate composition of the students in the district as a whole. Calculating enrollment and retention targets at the school district level is analogous to calculating school-level rates, and then applying differential weights to each school’s rate according to the number of students enrolled per school. Changes to the methodology: None. Distribution of ELLs Comment 6: Two comments stated that the proposed methodology ignores the historically diverse residency patterns of English language learners in New York State wherein speakers of other languages tend to settle in tight-knit neighborhoods rather than dispersing themselves throughout an arbitrarily large and disconnected geographic region. Suggested alternatives include: targets based on the demographics of the immediate neighborhood of each school rather than the school district or CSD at large. Discussion: The working group agrees, consistent with years of national research on immigration patterns, that English language learners in New York State do not reside evenly across school districts. However, the suggested approach would require establishing new borders for the purposes of defining enrollment and retention targets. Implementing a process to set new borders for the purpose of defining charter school enrollment and retention targets would be both costly and time consuming as a new data reporting structure would need to be created, and the nature of where and how those borders would be established would likely not be easily agreed upon. Changes to the Methodology: None. This comment will be considered during related accountability policy and practice development and implementation. Small “n” Size Samples Comment 7: One comment indicated that the statistical limitations on the retention targets that require at least 30 students be enrolled in an identified subgroup result no retention targets for the majority of New York City schools. Discussion: The statistical limitations on the original proposed methodology for generating enrollment and retention targets employ large sample inference methodologies. Thus, when sample sizes are small, the calculations do not meet standard statistical conventions. The working group was able to identify a similar small sample inference methodology (the Wilson Score Interval) that will allow for the calculation of retention targets in small sample size cases. Changes to the Methodology: To address the small sample size issues while continuing to allow an “error band” for the smallest schools and subgroups where there is the potential for higher levels of variability, the methodology will employ the Wilson Score Interval, a small sample inference technique, to generate confidence intervals and thereby effective targets. Whereas the confidence intervals generated from commonly employed large sample inference methods, such as the Wald method, rely on assumptions of normality in the data which are not valid across all schools in the target setting data files, the Wilson Score Interval employs valid statistical properties even when sample sizes are small. When sample sizes are larger, the Wilson Score Interval produces confidence intervals nearly identical to those generated from large sample inference methods. Existing Student Cohorts Comment 8: A comment suggested that the authorizers employ a phase-in approach to implementation whereby authorizers hold schools accountable for the enrollment demographics and retention statistics of new cohorts and not for those of existing cohorts. The comment suggested that it is impossible and unlawful for schools to implement any policies to adjust the enrollment demographics of existing cohorts, such as expelling students, but that schools can adjust the methods by which they attract particular students to entering cohorts. The comment suggested that the result would be a more fair comparison. Discussion: The working group agrees that charter schools cannot violate Education Law or applicable federal and state civil rights laws, and acknowledges that the Charter School’s Act requires schools to provide an admissions preference to returning students. Changes to the Methodology: None. This comment will be considered to guide related practice development and implementation. Charter Schools for Other At-risk Students Comment 9: One comment suggested that some charter schools function like “transfer high schools” in New York City. That is, some charter high schools indicate that part of their mission is to give preference to students who are at risk of failure in areas different from those identified in statute (foster care, homeless, juvenile justice, over-age, under-credited, etc). The comment suggests that these charter schools receive special consideration for retention targets, specifically, that authorizers compare them to the retention rates of transfer high schools in New York City. Discussion: The working group acknowledges that some charter schools were created with specific missions to serve at-risk students that may not align with the three at-risk categories defined in the statute and that some charter schools have existing admissions priorities for serving at-risk populations other than the three defined in the statute (students with autism, schools serving homeless students, student in foster care, student in the juvenile justice system, etc.). However, the practice of comparing a subset of charter schools to a subset of district schools misaligns with the language of the statute. As with the comment relating to the geographic distribution of ELL’s, the authorizers could take a particular school’s at-risk focus into account when determining whether a school has met or exceeded its targets, or whether the school has made good faith efforts to reach out to such identified communities Changes to the Methodology: None. This comment will be considered to guide related practice development and implementation. Blended Target Comment 10: Two comments stated that allowing a charter school to use a “blended target” to meet the requirements would be a violation of the letter and spirit of the law which clearly delineates three targets for three distinct categories of students. Discussion: Determinations of how schools will be held accountable for the targets will not be made until a final methodology is formally adopted by the Board of Regents and the SUNY Board of Trustees. Changes to the Methodology: None. Sub-categories of Targets Comment 11: One comment suggested that there should be targets set for sub-categories of English language learners and students with special education needs to ensure that charter schools are enrolling students at all levels of the ability spectrum in both categories. Discussion: The Board of Regents and the SUNY Board of Trustees could not, absent specific statutory authority, prescribe enrollment or retention targets separate from those identified in the Charter Schools Act. Currently, the Charter Schools Act only permits the authorizers to develop and prescribe enrollment and retention targets for the three identified at-risk designations, and if additional “sub-targets” were developed, schools could not legally be held accountable for failing to meet such targets. However, it should be noted that according to the proposed methodology, a student classified as at-risk in two or more categories will be tallied separately in each category for both traditional public schools and charter schools. Changes to the Methodology: None. Universal Free-Lunch Comment 12: Several comments indicated that the Universal Free Lunch program in New York City seemed to be skewing FRPL targets. They urged appropriate consideration in the final methodology to address this issue. Discussion: The final enrollment and retention targets for eligible applicants for the free and reduced price lunch program will not factor in participation in the New York City Department of Education’s Universal Free Lunch program. Only those students that have formally applied for and been awarded free or reduced price lunch will be tallied in developing the targets. Changes to the Methodology: None. Data Issues Comment 13: Several comments pointed out what was deemed to be inaccuracies in the provided empirical analysis files: a) source of data not clear as October 1 was not the BEDS date last year; b) there appear to be incorrect grade bands for a number of schools; c) there are a number of charter schools missing either enrollment and retention targets, more than just those that would fall into the 30 and under category; and d) there are a number of charter schools not included at all in the empirical analysis files. Discussion: The initial empirical files posted for contextual reference were based on a 2010-2011 school listing. In subsequent data runs, data has been updated to reflect the most current student-level records for 2011-2012 as well as the most current school listings for 2011-2012. Changes to the Methodology: None. Drawing Students from Multiple CSDs Comment 14: The proposed methodology does not seem to consider established charter schools that draw students from a number of different districts. Discussion: The working group understands that some schools may draw students from a number of different districts. However, the Charter School’s Act requires that the enrollment and retention targets be comparable to the charter school’s district of location (or Community School District of location in New York City). Change to the Methodology: None. Practice Comment 15: The final methodology should include a definition of what will be looked at by authorizers to determine “good faith effort” such as lottery pools; targeted recruitment methods; admissions preferences; and analyses that demonstrate that the charter school’s population reflects the school’s neighborhood more than the CSD as a whole. Discussion: This will be addressed in accountability policies developed by each authorizer as they determine how they will hold schools accountable. However, the Charter Schools Act requires, at minimum, that schools demonstrate that they have conducted outreach to parents and families in the surrounding communities, widely publicized the lottery for the school, and have in place programs to academically support each category of at-risk student for which there is a target. Changes to the Methodology: None. Comment 16: One comment noted that schools should be able to request a target adjustment when there have been major changes or enrollment shifts in their district or at their school. Discussion: The working group can envision limited instances where this would be appropriate within a given charter term. Significant enrollment changes and/or a move to a different district (or Community School District in New York City) would likely trigger such adjustments. Change to the Methodology: None. Processes for doing so could be considered by each authorizer in developing related practices for implementation. Policy Comment 17: One comment stated that establishing FRPL enrollment and retention targets propagates socioeconomic and racial segregation already established in district schools. Another comment indicated the same regarding ELL and SWD targets. Discussion: The work completed on enrollment and retention targets to date has been done to comply with the May 2010 amendments to the Charter Schools Act. Changes to the methodology: None. Comment 18: One comment indicated that documents shared with the proposed methodology indicate that each authorizer will be charged with creating standards to determine whether a school is making a good faith effort to recruit target populations. The comment stated that this will be particularly confusing in New York City for parents and advocates to ensure that appropriate process is being followed. The comment suggests NYSED and SUNY should develop standards, in collaboration with parents, schools and advocacy groups, which will be used by all authorizers to determine whether schools are making “good faith efforts” to enroll and retain these students. Discussion: To the extent possible, staff to the SUNY Trustees and the Board of Regents have worked cooperatively, and with the public, in developing the proposed target-setting methodology, and remain committed to involving the public in implementing enrollment and retention targets. The two governing boards oversee the charter schools that they are accountable for under independent contracts. Guiding policies and procedures, as well as authorizer actions related to enrollment and retention targets, may differ between governing boards when implemented. Changes to the methodology: None. Comment 19: Two comments noted that as charter schools begin to serve more students with disabilities and English language learners, authorizers need to ensure that appropriate programs are in place to serve these students and that school marketing materials clearly describe the available programs. Discussion: The working group appreciates the importance of the comment and notes that ensuring that schools have appropriate programs in place to serve students at risk of academic failure (in the three areas identified in the statute and beyond) is part of ongoing authorizer responsibility. The working group also notes that each authorizer currently reviews the academic effectiveness of each school’s at-risk program(s), and through city and state associations, additional professional development in serving students with disabilities and English language learners is being provided to NYS charter schools. Changes to the Methodology: None. 1 The target methodology was established by the New York State Education Department Charter School Office, in collaboration with the State University of New York Charter Schools Institute. 2 Type 2 discharges constitute all discharges other than a) articulation up to a higher school level, b) graduation, and c) death. 3 The use of one-sided confidence intervals is widespread among states for establishing accountability targets. See, for example: California - http://www.cde.ca.gov/ta/ac/ay/documents/aypinfoguide11.pdf; Missouri - http://dese.mo.gov/divimprove/sia/dar/documents/qs-si-understanding-your-ayp.pdf; and New Mexico - http://ped.state.nm.us/ayp2011/AYP%20FAQ%202011.pdf --------------- ------------------------------------------------------------ --------------- ------------------------------------------------------------ 2