A Report of the Assessment of Written Communication

A Report of the Assessment of Written Communication (AWC) 2021-2022 Description of Assessment of Written Communication (AWC) Each academic year, approximately 500 student writing artifacts are collected and assessed using a locally-developed writing rubric. This rubric was developed by faculty with expertise in teaching and assessing student writing and is assumed to have content related validity (Banta & Palomba, 2015). Over a three-year period, each academic college at SHSU will participate in the Assessment of Written Communication (AWC) and submit artifacts for scoring. These student artifacts either come directly from courses within those colleges or from required capstone projects; therefore, the artifacts represent authentic student work (Banta & Palomba, 2015; Kuh et al., 2015). The student data presented within this report reflect student performance regarding the Texas Higher Education Coordinating Board’s Core Learning Objective of Communication Skills (THECB, 2022). The THECB (2022) defines Communication Skills as “effective development, interpretation, and expression of ideas through written, oral and visual communication.” Data from this assessment may therefore be used to address the written communication element of the broader concept of Communication Skills. These data should be used in conjunction with other data to fully understand student knowledge and ability regarding this Core Learning Objective. Methodology A total of 301 artifacts from upper division courses in the College of Business Administration were scored as part of this writing assessment by faculty and staff volunteers during a two-day in-person scoring session in July 2022 using a locally-developed writing rubric. This rubric was divided into four separate domains: (1) Ideas/Critical Thinking/Synthesis; (2) Style; (3) Organization; and (4) Conventions. A copy of this rubric is provided in the Appendix. Each domain was scored individually from 1 to 4, with 1 being the lowest and 4 being the highest. Each artifact was reviewed by two raters, with a third rater introduced when the scores were too far out of agreement (i.e., a score of 1 and 4 for the same domain). The third rater would only score those domains that were not in agreement, and the two closest scores would be kept. The individual domain scores for each student writing artifact were then averaged together to provide a total average score for the artifact. Score Reliability Intraclass correlational coefficients (ICCs) were calculated to determine the level of interrater agreement for each domain of student writing, as well as the overall average scores (Fleiss, 2003; Shrout & Fleiss, 1979). According to Cicchetti (1994), ICC agreement values below .40 are to be interpreted as demonstrating poor agreement, from .40 to .59 as demonstrating fair agreement, .60 to .74 as demonstrating good agreement, and .75 and above as demonstrating excellent agreement. The agreement values for three of the four individual writing domains were in the good range, while Organization was fair, but approaching good. The overall average score was .76 indicating excellent agreement. A complete breakdown of the ICC agreement values can be found in Table 1. Table 1 Breakdown of ICC Agreement by Domain Area for the College of Business Administration Domain Area Ideas/Critical Thinking/Synthesis Style Organization Conventions Overall Average Intraclass Correlation for Average Measures .66 .68 .59 .67 .76 Results Descriptive statistics are provided of the average student score for each domain, as well as the overall average, for the College of Business Administration and its departments. Comparisons to previous data are also provided for the college and departments. The College of Business Administration was previously evaluated in 2018-2019. A full breakdown of collegelevel data can be found in Table 2. A breakdown of department-level data for the College of Business Administration can be found in Table 3. Table 2 Descriptive Statistics for Student Writing Performance for the College of Business Administration 2018-2019 AWC Scores College Ideas/Critical Thinking/Synthesis Style Organization Conventions Overall Average n 272 272 272 272 272 M 2.66 2.54 2.67 2.45 2.58 SD 0.65 0.68 0.68 0.74 0.61 2021-2022 AWC Scores n 301 301 301 301 301 M 2.67 2.60 2.75 2.61 2.65 SD 0.73 0.70 0.66 0.71 0.60 Table 3 Descriptive Statistics for Student Writing Performance by Department for Business Administration 2018-2019 AWC 2021-2022 AWC Scores Scores Department n n n n M SD Accounting Ideas/Critical Thinking/Synthesis 75 2.81 0.59 12 3.17 0.69 Style 75 2.68 0.66 12 3.04 0.69 Organization 75 2.89 0.64 12 3.17 0.58 Conventions 75 2.69 0.72 12 2.96 0.54 Overall Average 75 2.77 0.59 12 3.08 0.49 Economics and International Business Ideas/Critical Thinking/Synthesis 40 2.46 0.63 7 2.79 0.39 Style 40 2.51 0.60 7 2.64 0.75 Organization 40 2.55 0.72 7 2.93 0.67 Conventions 40 2.18 0.70 7 2.50 0.76 Overall Average 40 2.43 0.57 7 2.71 0.57 General Business and Finance Ideas/Critical Thinking/Synthesis 37 2.74 0.61 92 2.59 0.74 Style 37 2.49 0.72 92 2.47 0.70 Organization 37 2.62 0.64 92 2.66 0.67 Conventions 37 2.14 0.73 92 2.58 0.75 Overall Average 37 2.50 0.60 92 2.58 0.62 Management, Marketing, and Information Systems Ideas/Critical Thinking/Synthesis 120 2.60 0.68 190 2.67 0.72 Style 120 2.47 0.70 190 2.63 0.68 Organization 120 2.59 0.69 190 2.76 0.66 Conventions 120 2.50 0.71 190 2.60 0.70 Overall Average 120 2.54 0.62 190 2.66 0.59 Appendix Writing Assessment Rubric Writing Assessment Rubric This rubric asks you to identify features of the writing present in the sample. You should apply the numerical score based on degree of presence of the characteristic features. The writing features selected for the rubric are those most likely present in any disciplinary writing sample and represent a writing level expected of a senior-level college student. Legend: N/A = Not Applicable 1 = few features are present 2 = features are not often present 3 = features are often present 4 = features are most always present CATEGORY CHARACTERISTIC FEATURES Ideas/Critical Thinking/Synthesis • The depth of sophistication of thoughts and ideas. Features may include research, reasoning, evidence, detail, and development (appropriate to the field and genre) • • Central subject or argument of the assignment is easily identified, clearly emphasized, consistent with the evidence, and intriguing Reasoning is fully developed throughout the assignment with logical examples, details, and evidence where and as appropriate Assignment contains information that addresses counterarguments, biases, or reader’s expectations as appropriate Style • • • • • Sustained awareness of audience throughout the assignment Writing tone suits the audience and enhances the assignment’s purpose Sentence structure varies according to the content, purpose, and audience Sentences are consistently clear and logical Word choice is appropriate to the writing task Organization • • • • • Text is purposefully organized and substantially developed in a way that clarifies the argument and enhances style Arrangement of ideas (overall structure) is clear, logical, and compelling as appropriate to the assignment; the reader moves through the text easily Internal structure is cohesive and coherent; text flows and ideas are clearly and logically connected Transitions used appropriately Format is appropriate as defined by the assignment • • • • Grammar and mechanics support the reader’s understanding of the writer’s purpose without distracting errors Documentation style is consistent, if appropriate to assignment Sources, when appropriate, are effectively integrated into the body of the assignment Minor errors do not interfere with readability or damage the writer’s credibility (as appropriate to the assignment parameters) The choices the writer makes for specific audiences. Features may include word choice, tone, and sentence length and structure The coherence of the writing. Features may include balance and ordering of ideas, flow, transition, and appropriate format (as defined in assignment) Conventions Adherence to standard American edited English. Features include grammar, punctuation, capitalization, spelling, and documentation.