May 15th 2010

Report on the presentation at the Japan Society for Educational Technology:
"Study of the influence of modern character images appearing in test questions and giving oral encouragement to the test takers"

　We have experimented on the effects of modern character images uttering encouraging words to the test takers at the beginning and in the middle of tests. The results were orally presented at a research meeting of the Japan Society for Educational Technology held on the Asahikawa Campus of the Hokkaido University of Education.

　Most of the research concerning testing is centered on the study of the cognitive aspect. However, it is empirically shown that the sentiment of the examinee also affects test results.

　Many examinees tend to feel nervous or anxious while taking a test. If there is an opportunity to make them relax and ease their tension, some positive impact on the test results and their attitude towards test taking may be expected.

　In our research, we have used character images which appear in the middle of questions and speak to the examinee. Their effects on testing were studied.
　This is a follow-up research to the CRET research conducted in 2009 by Drs. Yuuki Kato, Shogo Kato, and Kanji Akahori as the Phase I study on the effects of character images giving encouragement.

　In our research as the Phase II, we have repeated the method used in the previous research. At the beginning and in the middle of the paper-based test, encouraging words coming from human characters were shown. We have examined whether this has helped ease the tension of the test takers and led to test taking in more relaxed manner, or whether this has had any negative impact. We have also studied if the encouragement by the character images has any impact on the actual implementation of the task.

　The following is the difference between the research of Phase I and Phase II..

　In the Phase I research, we set up three conditions and encouraging words which correspond to each three conditions, and compared them to each other. We used three characters to give encouraging words. One was a male junior high school student, the second was a female junior high school student, and the third was a pencil character. The first and second characters, which were human characters, seemed outdated and were seldom seen in teaching materials commercially available today.

　In the Phase II research, we used modern and trendy animated characters called "moe" in Japanese.

　In the Phase I research, examinees, who were university students, seemed to feel exposed to images of junior high school students because the characters were junior high school students. In the Phase II research, we utilized a mature teacher-like character. Conditions were increased by adding "no character" and "no encouraging words".

　We have analyzed the effects of each condition on the test results and the attitude of examinees. The result indicated that the encouragements given by the characters, in particular, have effectively influenced the way examinees tackled the problems.

　Compared with the Phase I research, the use of modern characters suggested the possibility of exerting a more favorable influence. As for the test results, there was no statistically significant difference. We have concluded that the encouragement by the characters helped students reduce the amount of wasted time and improved their attentiveness with more persistence during test taking.

　Our presentation invited many comments from the participants of the research meeting. They are summarized as follows;
　1. Since the use of characters giving encouragement had more impact on the way
　　 examinees carried on with their test taking, perhaps this method is considered more
　　 useful when used in various learning activities.
　2. Does the content of the test make a difference in the impact of the characters or not?
　3. The same type of research should be done with CBT as well.
　4. It is not possible to conclude that the examinees were more relaxed as a result of
　　 the encouragement by the characters.

　We intend to explore the open issues by taking account of these valuable comments.

May 15th 2010

Report on the presentation at the Japan Society for Educational Technology:
"Comparison of CBT and paper testing for regular expressions given to university students"

　An experiment was conducted to compare CBT and paper testing for regular expressions among university students and the result was orally presented at a research meeting of the Japan Society for Educational Technology held at the Asahikawa Campus of the Hokkaido University of Education.

　The objective of this research was to make a comparison between Computer Based Testing (CBT) and paper testing.

　In a field of learning such as regular expressions, respondents are able to simulate their answers in CBT. Therefore, the following assumptions were made for CBT.

　1. Test results can greatly improve compared against paper testing.
　2. Respondents are able to accurately predict their test results.
　3. The number of blank answers will increase.
　4. Respondents' impression after the test will improve.

　In order to verify this hypothesis, we have conducted a class on regular expressions for students from various universities. Forty students ranging from freshmen to seniors took a one-hour classroom lecture, immediately followed by testing. They were divided into two groups with twenty students in each. One group was given CBT and the other a paper test, each consisting of identical 12 questions. Before and after taking the test, each student was asked to respond to the questionnaire concerning the confidence for each question and the attributes of each student.

　The result of the experiment showed no significant difference in the test, which had a perfect score of 12 points. The CBT group scored 5.85 points, which was slightly higher than 5.05 points by the paper test group. As for the mean value of the difference between the expected and the actual score, there was no significant difference, with 1.50 points for CBT and 1.80 points for paper test group.

　A significant difference was shown in the rate of blank answers, with 4.58 items in the CBT group and 0.08 in the paper test group.

　We have concluded that in CBT on regular expressions, respondents tend to spend much time in their attempt to answer each question, and answer the questions only when they are confident and leave the rest blank. In other words, they are inclined to perfectionism and spend too much time on the questions in the beginning and tend to run out of time before answering the rest of the questions. As a result, the tendency is that their test results were not much different from those of the paper test group.

　In the Q & A session, there was a comment suggesting that the focus should be on a test which can be simulated, rather than on a comparison between CBT and a paper test.

October 10-12, 2009

Report on the JSSP 2009/JGDA 2009 (SP50) Presentation:
"The Concept of Teamwork: Working toward Measurement of Teamwork Capabilities"

　A presentation on the development of the scale to measure teamwork capabilities was given at JSSP2009/JGDA2009(SP50).

　A standardized scale to measure individual teamwork capabilities does not exist in Japan, even though several scales have been developed in other countries. There are three reasons which explain why relying on the translation of foreign research is not sufficient for the successful development of a teamwork scale for domestic use.

　1. A cultural gap may exist in the concept of teamwork.
　2. Depending on different conditions, such as team goals, activities, and members, a team
　　member may exhibit different qualities and contribute a different quantity of teamwork.
　3. Teamwork is a complicated phenomenon and around seven varieties of lower scale are
　　usually combined to measure teamwork in foreign countries.

　It is therefore fair to assume that the most reasonable combination of factors do vary, depending on the team goals, members, culture, etc.

　We have surveyed the connotation/meaning of the concept in order to find out what exactly it meant when Japanese university students, and workers in the U.S. and Japan, positively or negatively assessed teamwork.
　Based on its outcome and the existing related scale, 30 scale items which were deemed to reasonably measure teamwork (with high face validity) were selected. Japanese and English versions were created.
　A survey was conducted among Japanese and American workers to explore how each item is considered to effectively measure the degree of individual teamwork abilities. ("Face Validity Survey").

　The result of the connotation survey showed that Japanese university students frequently picked "cooperativeness" and "common goals". Japanese workers selected "fun", "ability to listen", "vigor", "openness", "accommodating", "laughter", "entrusting", "high" and other cheerful and enjoyable images associated with a work environment. On the other hand, in the U.S., there was a tendency to select words to fulfill the team mission such as "communication", "knowledge", "sharing", "leader", and "decision-making".
　Based on this observation, the Development of Testing Items Group at CRET has picked representative items from the existing scale items and integrated them into the newly developed items. Then, face validity was examined. Next, the gender and cultural gap between Japan and the U.S. was studied.

　An active Q&A session followed the presentation. Interest in future development, and the importance of "severity", "punishment", "compliance", which did not appear in the connotation survey, were suggested as important items during the Q&A.

October 10-12, 2009

Report on the JSSP 2009/JGDA 2009 (SP50) Presentation:
"Assessment of Shyness using IAT (2)"

　This is a report on the poster presentation given to the JSSP 2009/JGDA 2009 (SP50) on shyness assessment, using the Implicit Association Test (IAT). A fruitful discussion with many exchanges of opinions followed the presentation.

　Compared to other methods which measure individual differences in implicit cognition, IAT has been recognized as a test with superior reliability and stability. Since its advent, increasing numbers of research utilize IAT methodology. Asendorpf, Banse, & Mücke (2002) have validated the double dissociation between implicit and explicit personality self-concept in the case of shyness. In their model, the IAT predicts spontaneous shy behavior (tense posture, etc.), whereas the explicit ratings predict controlled shy behavior (length of speech), resulting in double dissociation.
　The research on implicit association measurement predicting the often uncontrollable behavior as represented in the observed change in performance of task achievement (e.g., Egloff & Schmukle, 2002; Fujii & Uebuchi, 2009) shows that it is possible to regard explicit measurement to predict self-controllable behavior, whereas implicit measurement predicts uncontrollable behavior.

　Considering the above research outcome, Fujii, Sugimori, & Aikawa (2008) developed IAT which measures shyness by referencing the Asendorpf et al. (2002) model, and implemented the test. The outcome was used to study the relationship between implicit shyness and social desirability. Additional data from Fujii and others was collected and used for re-analysis. The result is reported as Study 1. In Study 2, other assessments by friends and acquaintances of the Study 1 participants were used to examine the subject for assessment in shyness IAT.

October 27th, 2009

Report on E-Learn 2009 Presentation:
CBT R&D Report on "the Use of Animation to Display Math Problems in a Computer-Based Test"

　In late October 2009, we presented our research on the use of animation to display math problems in the computer-based testing interface, at the international conference, E-Learn (World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education 2009) held in Vancouver, Canada.

　This research was initiated to explore the alternative methods to measure math competence without relying on the examinees Ereading literacy. We created four types of math questions (solid figures, velocity, equation, pattern locus) in two versions: one version as animation and the other as written texts. The experiment was conducted involving 20 university students. Our major thrust of the experiment was to measure how well and how fast the examinees were able to understand the questions, rather than the accuracy of answers to the questions.

　Based on the questionnaire assessment, there was a significant difference in examinees Eimpression that the animation version seemed easier. However, based on the rubric assessment on understanding of questions, as well as the time required for comprehension of questions, there was no significant difference between the two versions. At E-Learn 2009, we have presented our conclusion by suggesting, as a possibility for math examinations, the use of animation instead of written questions.

　Three individuals posed questions and comments vis-à-vis our presentation, confirming the details of the experiment and analytical methods, as well as a comment that it would be a useful method of presenting questions to children with learning difficulties. Advice and positive feedback were received, such as ways to improve the item bank through animation, and its application to other science and math courses.

　This experiment may have depended on the characteristics of questions and the examinees, and therefore further verification is to take place.

October 24th, 2009

The influence of images of characters who speak encouraging words on examinees.

　Computer Based Tests (CBTs) are expected to be introduced in various scenes. CBTs may have fewer constraints than Paper Based Tests (PBTs) in terms of displaying the questions, and the methods of answering. Images and animation can also be used. Research was conducted as the first stage of examining how the use of character images may or may not influence examinees. By inserting human characters who speak encouraging words at the beginning or in the middle of a test, what kind of effects could be expected? Would it ease the tension of the examinees and help them relax, or might negative effects be expected instead? Would it affect the process of answering the questions? The study was carried out using paper-based experiments.

　Questions were created that were simple but rather monotonous, and could be irritating and time-consuming to answer. Images of characters were utilized at the beginning and in the middle of the tests. They spoke words meant to cheer up the test takers. Four conditions were used to differentiate the way words were spoken;

　1. The character of a boy spoke encouraging words.
　2. The character of a girl spoke encouraging words.
　3. An illustration of a pencil spoke encouraging words.
　4. Encouraging words were heard without character images.

　Results showed that using the human character images (1 and 2) negatively affected the performance and mood of test takers.

　Many comments were received after our report was presented at the Japan Society, which can be summarized as follows:

　1. There are a variety of character images that can be used. Therefore, a continued study should be made using different character images.
　2. The experiment used images of a boy and a girl. It may be interesting to compare the feedback based on how each gender responds to the character of their own gender or the opposite gender.
　3. The character images used looked like junior high school students. The examinees were university students. They may not have liked encouragement coming from younger characters.
　4. The test questions used were of a simple nature. If difficult questions were used, perhaps the results of the experiment might have been different.

　Based on the comments received, we intend to continue with our research.

October 24th, 2009

Research on the influence of "with" or "without" annotations on simple tests.

　Students generally use annotations while taking tests, such as underlining and marking key words, or scribbling in the margin of test sheets. It is possible with paper-based tests, but usually not with CBTs. If annotation gives positive effects on the test performance, CBTs need to be equipped with a comparable interface in order to avoid differences between CBTs and PBTs, caused by the ability or inability to annotate. If, however, there is no difference in performance, with or without annotation, there is no need to make an enhancement to CBTs with an interface for annotation.

　This research was conducted in a simple test to study the effect of annotation by comparing the results between two groups: one allowed to use annotation and the other not allowed to use it. The relationship between the examinees Eannotation activities in daily life and the control of annotation activities was also investigated.

　To our surprise, the result showed that there was little significance in the use of annotation. Annotation proved effective under the circumstances where annotation was absolutely advantageous. However, in general, there was almost no difference. When the test involved difficult reading or when annotation methods were not formalized, in some cases there was the possibility of annotation interfering with the answers. There are many open issues to be explored, such as the relationship between the difficulty of questions and the effects of annotation, needing to teach the right way to annotate or not, and the verification of the effect of teaching.

　In our presentation, we have also introduced the preceding research result. It showed that underlining while reading a book is effective, only when the text is quite difficult and there is sufficient time to do so. There was some feedback from the floor that it was surprising to hear that the effect of annotation was not very apparent. Most of the feedback consisted mainly of asking for the details of the research. There was not much discussion. We sensed that they had high expectations of further analysis. Our research did not include the analysis of the relationship between daily annotation activities and the test results of the university students who have participated in our experiment. We intend to announce the result in the future.

September 2nd, 2009

Report entitled "Methodology of Research on the University Teaching and its Application on the Freshmen Education (the First Year Experience at Universities and Colleges)" presented at the 5th National Convention on the Japan Association for Developmental Education.

　Dr. Kanji Akahori, a member of the CRET board of directors, has given the presentation at the 5th National Convention on the Japan Association for Developmental Education, held at Chitose Institute of Science and Technology from September 1st to 2nd, 2009.
　His report was based on the findings of the Design Based Research applied to analyze classroom teaching, design and assess the freshmen education at the universities and colleges.
　Here is the abstract of the presentation:

[Summary]
　Research methodology vis-à-vis university classroom teaching cannot be easily constructed in general. I have reviewed several research methods and decided to use the Design Based Research 2003, abbreviated as DBR, as the pragmatic and demonstrative approach. The report is on the findings based on the application of DBR on freshmen education.

　Since the direct application of DBR method accompanied difficulties, I have decided to use the revised version which was developed as a method to improve university classroom teaching. Using the revised DBR, an analysis, design and assessment of classroom teaching was made.

　As part of the freshmen education, I teach a subject named freshmen seminar. The content of the course consists of the common freshmen curriculum, flavored with newly developed content through the ingenuity of the faculty members.

　The outline of the freshmen education consists of basic courses such as orientation, self-introduction and a campus tour. I have added specific content such as writing, newspaper reading, educational counseling, basic education terminology, note-taking, and logical writing into the freshmen courses to complement each other. The content is comparable to the seminar offered by the Department of Education.

　The DBR and the revised DBR enable us to repeat the designing process while assessing each program, and explain it using past findings in addition to generating new methodologies as findings.
　In this research, I shall report on the findings gained by applying the revised DBR.

August 5, 2009

Report entitled "Research on Vocabulary Competency for school subjects and PISA-type Knowledge Application Competency"
presented at the 37th Convention of the Behaviormetric Society of Japan

　A research group in the Assessment and Analysis of Educational Testing section has been engaged in the research and development of multiple-choice testing to measure the mastery of vocabulary for key school subjects. This is part of a more general project which is intended to develop methods to predict mastery of school subjects in simpler manners.
　In 2008, the research group developed vocabulary tests for four key subjects (Japanese, arithmetic/mathematics, science, and social studies) and a PISA-type knowledge application competency test. A preliminary survey was conducted targeting fifth graders and eighth graders. The result was presented at the 37th Convention of the Behaviormetric Society of Japan.

　Many scholars from universities attended the session. One of the questions asked was on the recommended teaching method for students who scored low on vocabulary tests. Another question was whether the unidimensionality of each vocabulary test had been verified by subject area.

March 11th, 2009

The CBT Interface Research Development Report on the Intensive Reading Comprehension Exam: ED-MEDIA 2008 Presentation Report

Scene of the Experiment

System Display

Presentation at ED-MEDIA

　At the Research Department on Advancement of Testing Technology, we are currently researching a new test interface as a part of a motherboard development project for next-generation examinations. In 2008, we researched and developed a test interface for an intensive reading comprehension exam that is written vertically. After conducting an experiment with 22 university students as test subjects, a presentation was given in the Short Paper (New Development) section at the international ED-MEDIA conference.

　When vertically-written Japanese is displayed on a computer screen, which is often the case for intensive reading comprehension exams, the entire examination text, using this new interface, can be viewed in a single screenshot. The new interface can also display the same number of letters per vertical line as exams printed on paper. Moreover, this interface can be used to display a variety of printing formats commonly used in various media, including the 41-letter (per vertical line) format in the case of the new pocket paperback, 20 letters in the case of manuscript paper, and 13 letters in the case of newsprint. The display can be changed by zooming and scrolling with the mouse, giving the examinee the ability to view the entire exam in one glance. This has enabled us to create a test environment similar to that of an exam printed on paper.

　Although the focus of this presentation was on the development of the interface for the purpose of reading or writing Japanese, it was seen as something other languages could benefit from. Three researchers from English-speaking countries asked us questions and made suggestions. We exchanged opinions about the fact that students examined on printed materials scored higher. We also discussed the algorithm used in enlarging font sizes as well as its implementation methods; the discussion was useful for future research. We will continue to make suggestions for new methods of presenting examination questions, as well as researching the effects that writing long answers on a computer has on examinees. We also hope to further our research on how to deal with the issue of showing the intermediate steps taken by the examinee when solving a mathematical problem. Finally, we hope to create a test environment that links the above to an automatic marking system.