Fine Arts Assessment Pilot Report

 
The Missouri Alliance for Arts Education developed a project to study the effectiveness of implementing a web-based fine arts assessment to fifth-grade students in Missouri public schools. Ten music and art teachers from six school districts agreed to participate in the study that surveyed 364 students. The pilot was conducted May 1-5, 2012.

The Fine Arts Assessment Pilot included ten selected response items (multiple-choice) and 3 constructed-response items taken from the Missouri Department of Elementary and Secondary Education (DESE) Fine Arts Component of the Missouri Assessment Program (Fine Arts MAP). 

The original Fine Arts MAP was piloted in 2000 and implemented as a voluntary assessment in more than 300 school districts across the state in 2001. The assessment was on track to be included in the complement of assessments used to determine school accreditation until funding was withdrawn in August 2001. The Fine Arts MAP developed by DESE had 40 selected-response items and 5 constructed-response items covering content in music, visual art, theatre and dance. 

The format of the original Fine Arts MAP included video and audio delivered via VHS tape. During Section 1, students viewed artwork or listened and watched music, theatre or dance prompts which were followed by a narrator reading the question and the four answer choices. The students were instructed to watch or listen to the selection a second time and then record the answer in the test booklet. During Section 2, the constructed-response section, students viewed and/or listened to prompts, the item was read by the narrator, the prompt was repeated and students wrote answers in the test booklets. 

The MAAE converted the video to a digital format and developed a secure, password-protected website accessible to teachers who administered the assessment. Teachers could request printed test booklets from the MAAE or could print the booklets themselves. The student and teacher surveys were also provided or printed by the participants. Four of the six school districts chose to print the test booklets and surveys. All of the schools returned the surveys at their own expense.

The purpose of the study was to collect data about administering the web-based assessment using a smart board or similar technology. The MAAE surveyed students and teachers regarding the experience. Student knowledge of fine arts content was not part of the pilot; therefore student responses to the 13 items on the test were not scored. Teachers who chose to score their students’ responses were provided scoring guides to assist them. 

The MAAE did not collect student data regarding the number of correct responses. Student proficiency could not be determined in the pilot since using the number of correct responses is not a valid or reliable method of determining achievement levels for this assessment. To set achievement levels, the web-based assessment would have needed to include the 45 items from the original FA MAP.   

The Results: When asked about the level of difficulty of the 13 items, 63% of the students surveyed indicated the test was “somewhat easy” and 21% rated the test as “easy.” Thirteen percent of the students said the test was “somewhat difficult” and 3% of the students said the items were “difficult.” It is important to note that since achievement-level data was not a goal of the study, the test items were selected based on the quality of the video and audio as well as the inclusion of all four arts disciplines.

Although the comments written by students often indicated too much time was given to answer the questions, 67% of those surveyed said the amount of time was “about right;” twenty-two percent said the time allowed was “too much ”and 11% said it was “too little.” When the original Fine Arts MAP was developed, a variety of response times were studied to determine the final amount of time. In the twelve years since those studies, most students have been exposed to faster-paced online experiences which may change results of future studies in the amount of time needed for student responses. 

Fifty-eight percent of the students said the narrated instructions helped them understand when they should be looking at the video, when they should be looking at the test book, and when to answer the test questions; 34% said the instructions were “somewhat” helpful while 3% of the students indicated the instructions were not necessary.

Half of the students surveyed indicated that having the questions read aloud helped them to understand them better than if they read them from a test book while another 26% said that it was “sometimes” helpful; slightly less than a quarter (24%) of the students said they did not need to have the questions read aloud.

One of the advantages of the format of the Fine Arts MAP is that students who ordinarily need special accommodations during MAP testing are able to take the test with little assistance. The narrated questions and answers as well as the significant amount of time allowed to select responses add to the equitable nature of the test developed by the State. Only one percent of the students who took the test in the MAAE pilot used laptop computers with assistance from teachers.

A concern when delivering fine arts video and audio in a large-screen format is the ability of the students to see and hear the content. Eighty percent of the students said they could hear the audio clearly and 13% selected the “somewhat” response while 7% had difficulty hearing the audio. One teacher indicated high-quality external computer speakers helped his students hear the audio without difficulty. Seventy-eight percent of the students said they could see and understand the artworks and performances clearly and 22% said they could not. The teachers had the option to administer the test in a computer lab on individual computers however all chose to use the large-screen format. 

The following comments written by students regarding the method of administering the test may be helpful in designing future projects.
  • Students should be able to take the test by themselves.
  • It was too slow for me and I felt like it took a long time. I think we should have done it at our pace not the video’s
  • If you’re worried about the time then make it a little shorter; teachers can pause.
  • I think it was a good idea to do the question read aloud because sometimes if you read them in your head you can’t understand them.
  • I think it should show on the screen how much time you have left.
  • I didn’t like having only 2 minutes for the writing. Also I think we should have more about acting and dance.

Based on the surveys and the student comments, the MAAE may determine strategies for improving future studies of fine arts assessment. In reviewing the process for the pilot it would have been beneficial to have had a conference call with all teachers who were planning to administer the test to increase teacher understanding of the purpose of the pilot. Since school district approval and building-level administrator approval was the first step in selecting schools for the pilot, most of the teachers did not have direct contact with the MAAE project manager. Only half of the teachers submitted surveys following the pilot which indicated that they may not have realized the surveys were the most important factor in the study results.  All of the teachers submitted the student surveys which were also an important part of the study.

Studies have shown that the positive or negative attitude of the teacher administering the test affects student responses. It would also have been helpful to have provided a script for teachers to follow when administering the test. There appeared to have been a correlation between negative comments from students of teachers who submitted negative comments on surveys or who did not complete a survey. In one class, multiple students used the term “worthless” when describing the test. That word choice did not appear in any other school or class. 

The advantage of a web-based administration is that the test is cost effective. The cost to develop the video and audio prompts has been reduced over the past decade due to readily-available recording and editing equipment to non-professionals. The cost of creating secured websites is also lower. One hundred percent of the schools participating in the pilot were able to access the video on the website using one of three formats provided. 

Recommendations for future fine arts assessment projects include soliciting examples of teacher-created traditional assessments, performance-based assessments and portfolio assessments as well as updating and improving the concept of a web-based arts assessment.