Monthly Archives: May 2013

How Do You Know When It’s Good?

I have always been a decent student.  I was in National Honor Society in high school and graduated cum laude from college.  To date, I have a 4.0 in my graduate studies.  But what do all those honors mean in terms of what I’ve learned?  And what should my take aways be from my academic achievements now that the student is becoming the instructional designer?

Over the past several years, I’ve written about evaluation and assessment one, two, three times at least.  It’s a topic I enjoy revisiting, so it seemed appropriate that I choose this question to answer with my final reflective blog for this particular course – What is the standard process for project evaluation?

My short answer is – there is no standard process.  Or an even shorter answer – it depends.

My long answer goes like this.  There are countless ways to look at a program in terms of its effectiveness: Are there measurable learning objectives? Is the content appropriate to achieve the objectives? How is technology used to enhance learning?  Is there a logical flow between modules?  Are the choices of assessment tools suitable to measure the learning objectives?  Is tech support available when needed?  How is formative assessment used to improve the learning during the course?  How is summative assessment used (if an assessment is given as an end result and never viewed with an eye toward how it can be used to improve learning or teaching, has there been a bigger opportunity missed?)? Is the assessment authentic (meaning that it demonstrates you understand “both the products and the processes of learning.”)?

For every question, there are myriad resources available to provide guidance to the struggling new instructional designer.  One of the most often-cited tools is the Chico Rubric for Online Instruction (ROI) which is designed to answer the question, “What does high-quality online instruction look like?”  Far be it from me to tear down a highly esteemed tool, but I’m not a huge fan of the ROI.

With the Chico ROI, the three rankings seem to start from the assumption that the lowest ranking is acceptable by using the term baseline.  However, if you read some of the descriptions for the baseline level, you’ll see phrases such as “not aligned to learning objectives” and “limited or no activities to help students develop critical thinking.”  Other baseline rankings feel more appropriate to be deemed a starting point such as “New teaching methods are applied to enhance student learning” and “Assessment strategies are used to measure content knowledge, attitudes and skills.”  It is difficult to determine if a baseline ranking is really all that bad or just allows for improvement.

So I went on the hunt for other program evaluation tools, and I discovered that even experienced learning professionals are looking for a checklist or rubric that can be used to assess the quality of the instructional design.  Like the Chico ROI, many resources are free to use; however, I did discover one that cost $500 on top of a Continuing and Professional Education membership that cost over $3,000!  For that much money, that rubric ought not only to provide a score but also a 20 page report that details suggestions for improvement!  I kept hunting…

I found tools that simply captured YES/NO answers, offered guidance for specific letter grades, and one that used a spectrum between “action oriented materials” and “information dump.”

Spectrum rubric

After hours of research, I felt no closer to finding a standard process for how to evaluate a program’s effectiveness than when I started.  At this point, I turned my search inward and asked myself if the classes I’ve been taking recently have been good, and if so, how do I know that?  Call it intuition or maybe even a gut-check rubric, but I can say without a doubt that – at least for me – these classes have provided some phenomenal learning opportunities.

Was it because the learning objectives and instructional and assessment activities were closely aligned?  Maybe.  Did the courses offer ample opportunities for interaction and communication student-to-student, student-to-instructor, and student-to-content?  Yes, but it was more than that.  I know good learning is happening because, just as I did when I wrote my final reflection for ADLT 606, “as I sit down to write this reflective essay, I am thinking back not only on what I’ve learned in this course, but on how what I’ve learned in this course has brought up some of the themes I’ve been thinking about over the last two [now three] years.”  They call that transfer of learning, right?

While one of my classmates accurately pointed out that “sometimes what is modeled doesn’t transfer until the course is long over,” I’ve been fortunate to find ways to model what I’m learning now as well as revisit what I’ve learned before and incorporate it into my new learning.  My efforts have been acknowledged by my professors, but (and I mean no disrespect) it’s been my own words that have rewarded me by showing how much I have learned over the past few years and how well I weave my academic learning into every aspect of my life.

So for me, it is not so much transferring what I learned into my own teaching practice right now as it is about paying attention to these practices in other areas – such as the recent trend in gamification in marketing, the move toward mobile technologies, and questions surrounding analytics to assess ROI on social media – and applying what I have learned in this class to understand how to answer the questions that surround these topics in a different context. (ADLT 640 Final Self-Assessment)

Through self-assessments and especially through my reflective blogging practice, I am proving to myself that I am establishing significant learning.


Posted by on May 8, 2013 in ADLT 642