7 Reporting and Results | 2023–2024 Technical Manual Update

7.1 Student Participation

During 2023–2024, assessments were administered to 21,526 students in seven states. Table 7.1 displays counts of students tested in each state. The assessments were administered by 6,307 educators in 4,795 schools and 1,317 school districts during the two instructionally embedded windows. A total of 522,016 test sessions were administered across both assessment windows. One test session is one testlet taken by one student. Only test sessions that were complete at the close of each testing window counted toward the total sessions.

Table 7.1: Student Participation by State in 2023–2024 (N = 21,526)
State	Students (n)
Arkansas	2,716
Delaware	778
Iowa	2,975
Kansas	2,378
Missouri	4,573
North Dakota	618
Tennessee	7,488

Table 7.2 summarizes the number of students assessed in each grade. In grades 3–8, over 2,800 students participated in the DLM assessment at each grade. In high school, the largest number of students participated in grade 11, and the smallest number participated in grade 12. The differences in high school grade-level participation can be traced to differing state-level policies about the grade(s) in which students are assessed.

Table 7.2: Student Participation by Grade in 2023–2024 (N = 21,526)
Grade	Students (n)
3	2,808
4	2,971
5	2,890
6	2,990
7	2,882
8	2,836
9	807
10	1,208
11	2,108
12	26

Table 7.3 summarizes the demographic characteristics of the students who participated in the 2023–2024 administration. The majority of participants were male (67%), White (67%), and non-Hispanic (88%). About 7% of students were monitored or eligible for English learning services.

Table 7.3: Demographic Characteristics of Participants in 2023–2024 (N = 21,526)
Subgroup	n	%
Gender
Male	14,384	66.8
Female	7,142	33.2
Race
White	14,506	67.4
African American	4,816	22.4
Two or more races	1,129	5.2
Asian	623	2.9
American Indian	302	1.4
Native Hawaiian or Pacific Islander	139	0.6
Alaska Native	11	0.1
Hispanic ethnicity
Non-Hispanic	18,892	87.8
Hispanic	2,634	12.2
English learning (EL) participation
Not EL eligible or monitored	19,995	92.9
EL eligible or monitored	1,531	7.1

7.2 Student Performance

Student performance on DLM assessments is interpreted using cut points determined by a standard setting study.⁸ For a description of the standard setting process used to determine the cut points, see Chapter 6 of this manual. Student achievement is described using four performance levels. A student’s performance level is determined by the total number of linkage levels mastered across the assessed Essential Elements (EEs).

For the 2023–2024 administration, student performance was reported using four performance levels:

The student demonstrates Emerging understanding of and ability to apply content knowledge and skills represented by the EEs.
The student’s understanding of and ability to apply targeted content knowledge and skills represented by the EEs is Approaching the Target.
The student’s understanding of and ability to apply content knowledge and skills represented by the EEs is At Target. This performance level is considered meeting achievement expectations.
The student demonstrates Advanced understanding of and ability to apply targeted content knowledge and skills represented by the EEs.

7.2.1 Overall Performance

Table 7.4 reports the percentage of students achieving at each performance level on the 2023–2024 ELA and mathematics assessment administration by grade and subject. While the high school mathematics blueprints organize EEs by grade level, the high school ELA blueprints are based on grade bands (e.g., 9–10 and 11–12) that mirror the organization of the EEs. In ELA, the percentage of students who achieved at the At Target or Advanced levels (i.e., proficient) ranged from approximately 12% to 39%. In mathematics, the percentage of students who achieved at the At Target or Advanced levels ranged from approximately 8% to 19%.

Table 7.4: Percentage of Students by Grade and Performance Level
Grade	n	Emerging (%)	Approaching (%)	At Target (%)	Advanced (%)	At Target + Advanced (%)
English language arts
3	2,807	40.0	27.1	27.9	5.0	32.9
4	2,968	37.4	26.1	32.4	4.0	36.5
5	2,885	28.9	32.5	32.1	6.5	38.6
6	2,985	33.6	35.3	25.8	5.2	31.0
7	2,881	36.5	37.1	20.5	5.8	26.3
8	2,831	51.7	23.8	19.0	5.5	24.5
9-10	2,014	59.6	28.4	9.6	2.4	12.0
11-12	2,131	49.4	32.5	11.7	6.4	18.1
Mathematics
3	2,801	61.3	31.1	5.9	1.7	7.6
4	2,963	55.6	31.7	9.3	3.4	12.7
5	2,870	56.1	31.5	9.2	3.2	12.4
6	2,976	66.0	26.2	5.5	2.3	7.8
7	2,876	64.2	24.3	7.9	3.6	11.5
8	2,823	63.3	23.8	10.1	2.8	12.9
9	802	52.4	30.8	10.8	6.0	16.8
10	1,199	33.4	47.5	16.3	2.8	19.1
11	2,099	49.7	37.1	10.0	3.1	13.1

7.2.2 Subgroup Performance

Data collection for DLM assessments includes demographic data on gender, race, ethnicity, and English learning status. Table 7.5 and Table 7.6 summarize the disaggregated frequency distributions for ELA and mathematics performance levels, respectively, collapsed across all assessed grade levels. Although state education agencies each have their own rules for minimum student counts needed to support public reporting of results, small counts are not suppressed here because results are aggregated across states and individual students cannot be identified.

Table 7.5: ELA Performance Level Distributions by Demographic Subgroup in 2023–2024 (N = 21,502)
	Emerging		Approaching		At Target		Advanced		At Target + Advanced
Subgroup	n	%	n	%	n	%	n	%	n	%
Gender
Male	5,965	41.5	4,343	30.2	3,334	23.2	727	5.1	4,061	28.3
Female	2,875	40.3	2,193	30.7	1,681	23.6	384	5.4	2,065	28.9
Race
White	5,922	40.9	4,305	29.7	3,444	23.8	819	5.7	4,263	29.4
African American	2,011	41.8	1,523	31.7	1,082	22.5	192	4.0	1,274	26.5
Two or more races	442	39.1	373	33.0	261	23.1	53	4.7	314	27.8
Asian	292	46.9	190	30.5	118	18.9	23	3.7	141	22.6
American Indian	111	36.8	96	31.8	77	25.5	18	6.0	95	31.5
Native Hawaiian or Pacific Islander	60	43.2	46	33.1	28	20.1	5	3.6	33	23.7
Alaska Native	2	18.2	3	27.3	5	45.5	1	9.1	6	54.5
Hispanic ethnicity
Non-Hispanic	7,663	40.6	5,706	30.2	4,490	23.8	1,010	5.4	5,500	29.1
Hispanic	1,177	44.7	830	31.5	525	19.9	101	3.8	626	23.8
English learning (EL) participation
Not EL eligible or monitored	8,233	41.2	6,022	30.2	4,665	23.4	1,052	5.3	5,717	28.6
EL eligible or monitored	607	39.7	514	33.6	350	22.9	59	3.9	409	26.7

Table 7.6: Mathematics Performance Level Distributions by Demographic Subgroup in 2023–2024 (N = 21,409)
	Emerging		Approaching		At Target		Advanced		At Target + Advanced
Subgroup	n	%	n	%	n	%	n	%	n	%
Gender
Male	8,265	57.8	4,285	30.0	1,288	9.0	463	3.2	1,751	12.2
Female	4,173	58.7	2,175	30.6	584	8.2	176	2.5	760	10.7
Race
White	8,220	57.0	4,408	30.6	1,318	9.1	479	3.3	1,797	12.5
African American	2,916	60.9	1,407	29.4	374	7.8	95	2.0	469	9.8
Two or more races	652	58.1	348	31.0	89	7.9	34	3.0	123	11.0
Asian	404	65.2	151	24.4	52	8.4	13	2.1	65	10.5
American Indian	155	51.8	102	34.1	28	9.4	14	4.7	42	14.0
Native Hawaiian or Pacific Islander	87	62.6	40	28.8	9	6.5	3	2.2	12	8.6
Alaska Native	4	36.4	4	36.4	2	18.2	1	9.1	3	27.3
Hispanic ethnicity
Non-Hispanic	10,811	57.5	5,728	30.5	1,681	8.9	566	3.0	2,247	12.0
Hispanic	1,627	62.0	732	27.9	191	7.3	73	2.8	264	10.1
English learning (EL) participation
Not EL eligible or monitored	11,552	58.1	6,006	30.2	1,744	8.8	583	2.9	2,327	11.7
EL eligible or monitored	886	58.1	454	29.8	128	8.4	56	3.7	184	12.1

7.3 Mastery Results

As previously described, student performance levels are determined by applying cut points to the total number of linkage levels mastered in each subject. This section summarizes student mastery of assessed EEs and linkage levels, including how students demonstrated mastery from among three scoring rules and the highest linkage level students tended to master.

7.3.1 Mastery Status Assignment

As described in Chapter 5 of the 2021–2022 Technical Manual—Instructionally Embedded Model (Dynamic Learning Maps Consortium, 2022), student responses to assessment items are used to estimate the posterior probability that the student mastered each of the assessed linkage levels using diagnostic classification modeling. The linkage levels, in order, are Initial Precursor, Distal Precursor, Proximal Precursor, Target, and Successor. A student can be a master of zero, one, two, three, four, or all five linkage levels, within the order constraints. For example, if a student masters the Proximal Precursor level, they also master all linkage levels lower in the order (i.e., Initial Precursor and Distal Precursor). Students with a posterior probability of mastery greater than or equal to .80 are assigned a linkage level mastery status of 1, or mastered. Students with a posterior probability of mastery less than .80 are assigned a linkage level mastery status of 0, or not mastered. Maximum uncertainty in the mastery status occurs when the probability is .5, and maximum certainty occurs when the probability approaches 0 or 1. In addition to the calculated probability of mastery, students could be assigned mastery of linkage levels within an EE in two other ways: correctly answering 80% of all items administered at the linkage level or through the two-down scoring rule. The two-down scoring rule was implemented to guard against students assessed at the highest linkage levels being overly penalized for incorrect responses. When a student did not demonstrate mastery of the assessed linkage level, mastery was assigned at two linkage levels below the level that was assessed. If a student tested at more than one linkage level for the EE and did not demonstrate mastery at any level, the two-down rule was applied according to the lowest linkage level tested. The two-down rule is based on linkage level ordering evidence and the underlying learning map structure which is presented in Chapter 2 of the 2021–2022 Technical Manual—Instructionally Embedded Model (Dynamic Learning Maps Consortium, 2022).

As an example of the two-down scoring rule, take a student who tested only on the Target linkage level of an EE. If the student demonstrated mastery of the Target linkage level, as defined by the .80 posterior probability of mastery cutoff or the 80% correct rule, then all linkage levels below and including the Target level would be categorized as mastered. If the student did not demonstrate mastery on the tested Target linkage level, then mastery would be assigned at two linkage levels below the tested linkage level (i.e., mastery of the Distal Precursor), rather than showing no evidence of EE mastery at all.

The percentage of mastery statuses obtained by each scoring rule was calculated to evaluate how each mastery assignment rule contributed to students’ linkage level mastery statuses during the 2023–2024 administration of DLM assessments (see Figure 7.1). Posterior probability was given first priority. That is, if scoring rules agreed on the highest linkage level mastered within an EE (i.e., the posterior probability and 80% correct rule both indicate the Target linkage level as the highest mastered), the mastery status was counted as obtained via the posterior probability. If mastery was not demonstrated by meeting the posterior probability threshold, the 80% scoring rule was imposed, followed by the two-down rule. This means that EEs that were assessed by a student at the lowest two linkage levels (i.e., Initial Precursor and Distal Precursor) are never categorized as having mastery assigned by the two-down rule. This is because the student would either master the assessed linkage level and have the EE counted under the posterior probability or 80% correct scoring rule, or all three scoring rules would agree on the score (i.e., no evidence of mastery), in which case preference would be given to the posterior probability. Across grades and subjects, approximately 81%–92% of mastered linkage levels were derived from the posterior probability obtained from the modeling procedure. Approximately 2%–8% of linkage levels were assigned mastery status by the percentage correct rule. The remaining 6%–11% of mastered linkage levels were determined by the two-down rule.

Figure 7.1: Linkage Level Mastery Assignment by Mastery Rule for Each Subject and Grade

Two sets of stacked bar charts for ELA and mathematics. There is a bar chart for each grade, and the stacks within each bar chart represent a mastery rule and the percentage of mastery statuses obtained by each scoring rule. The highest percentage of linkage level mastery assignment across all grades is for the posterior probability mastery rule.

Because correct responses to all items measuring the linkage level are often necessary to achieve a posterior probability above the .80 threshold, the percentage correct rule overlaps considerably with the posterior probabilities (but is second in priority). The percentage correct rule did provide mastery status in instances where correctly responding to all or most items still resulted in a posterior probability below the mastery threshold. The agreement between the posterior probability and percentage correct rules was quantified by examining the rate of agreement between the highest linkage level mastered for each EE for each student using each method. For the 2023–2024 operational year, the rate of agreement between the two methods was 86%. When the two methods disagreed, the posterior probability method indicated a higher level of mastery (and therefore was implemented for scoring) in 67% of cases. Thus, in some instances, the posterior probabilities allowed students to demonstrate mastery when the percentage correct was lower than 80% (e.g., a student completed a four-item testlet and answered three of four items correctly).

7.3.2 Linkage Level Mastery

Scoring for DLM assessments determines the highest linkage level mastered for each EE. This section summarizes the distribution of students by highest linkage level mastered across all EEs. For each student, the highest linkage level mastered across all tested EEs was calculated. Then, for each grade, the number of students with each linkage level as their highest mastered linkage level across all EEs was summed and then divided by the total number of students who tested in the grade and subject. This resulted in the proportion of students for whom each level was the highest linkage level mastered.

Figure 7.2 displays the percentage of students who mastered each linkage level as the highest linkage level across all assessed EEs for ELA and mathematics. For example, across all grade 3 ELA EEs, the Distal Precursor level was the highest level that 42% of students mastered. The percentage of students who mastered the Target or Successor linkage level as their highest level ranged from approximately 15% to 41% in ELA and from approximately 10% to 20% in mathematics.

Figure 7.2: Students’ Highest Linkage Level Mastered Across English Language Arts and Mathematics Essential Elements by Grade in 2023–2024

Two sets of stacked bar charts for ELA and mathematics. There is a bar chart for each grade, and the stacks within each bar chart represent a linkage level and the percentage of students who mastered that linkage level as their highest level. The highest linkage level for most students was below the Target level.

7.4 Data Files

DLM assessment results were made available to DLM state education agencies following the 2023–2024 administration. Similar to previous years, the General Research File (GRF) contained student results, including each student’s highest linkage level mastered for each EE and final performance level for the subject for all students who completed any testlets. In addition to the GRF, the states received several supplemental files. Consistent with previous years, the special circumstances file provided information about which students and EEs were affected by extenuating circumstances (e.g., chronic absences), as defined by each state. State education agencies also received a supplemental file to identify exited students. The exited students file included all students who exited at any point during the academic year. In the event of observed incidents during assessment delivery, state education agencies are provided with an incident file describing students affected; however, no incidents occurred during 2023–2024.

Consistent with previous delivery cycles, state education agencies were provided with a 2-week window following data file delivery to review the files and invalidate student records in the GRF. Decisions about whether to invalidate student records are informed by individual state policy. If changes were made to the GRF, state education agencies submitted final GRFs via Educator Portal. The final GRF was used to generate score reports.

7.5 Score Reports

Assessment results were provided to state education agencies to report to parents/guardians, educators, and local education agencies. Individual Student Score Reports summarized student performance on the assessment by subject. Several aggregated reports were provided to state and local education agencies, including reports for the classroom, school, district, and state.

No changes were made to the structure of individual or aggregated reports during spring 2024. For a complete description of score reports, including aggregated reports, see Chapter 7 of the 2021–2022 Technical Manual—Instructionally Embedded Model (Dynamic Learning Maps Consortium, 2022).

7.5.1 Individual Student Score Reports

Similar to previous years, Individual Student Score Reports included two sections: a Performance Profile, which describes student performance in the subject overall, and a Learning Profile, which provides detailed reporting of student mastery of individual skills. During 2023–2024, a new helplet video was created to support interpretation of score reports. For a description of the new score report interpretation video, see Chapter 9 of this manual. Further information on evidence related to the development, interpretation, and use of Individual Student Score Reports and sample pages of the Performance Profile and Learning Profile can be found in Chapter 7 of the 2021–2022 Technical Manual—Instructionally Embedded Model (Dynamic Learning Maps Consortium, 2022).

7.6 Quality-Control Procedures for Data Files and Score Reports

No changes were made to the quality-control procedures for data files and score reports for 2023–2024. For a complete description of quality-control procedures, see Chapter 7 of the 2021–2022 Technical Manual—Instructionally Embedded Model (Dynamic Learning Maps Consortium, 2022).

7.7 Conclusion

Results for DLM assessments include students’ overall performance levels and mastery decisions for each assessed EE and linkage level. During 2023–2024, ELA and mathematics assessments were administered to 21,526 students in seven states adopting the Instructionally Embedded model. Between 8% and 39% of students achieved at the At Target or Advanced levels across all grades and subjects. Of the three scoring rules, linkage level mastery status was most frequently assigned by the posterior probability of mastery, and students tended to demonstrate mastery of the Target or Successor level at higher rates in English language arts than in mathematics.

Following the 2023–2024 administration, three data files were delivered to state education agencies: the GRF, the special circumstance code file, and the exited students file. No changes were made to the structure of data files, score reports, or quality-control procedures during 2023–2024.