Musings about level 1 evaluation....

While I'm not much of a fan of Level I evaluation, I do think it can shed insight into the effectiveness of our programs if we ask the right questions and pay attention to the answers. I'm dealing with metrics-fans right now who want to ask smile-sheet questions like, "On a scale of 1 to 6, did you find the training useful? ..." What am I supposed to do with knowing that people ranked the training as an average of 5.9 in 'usefulness'? Or worse, a 2.6?



Here's an evaluation Kassy LaBorie and I did yesterday in the wrap up to our online "Games Synchronous Trainers Play" session. (See citation at the bottom of this post). It tells us much more than the typical "smile sheet"



What can I tell from this? That we emphasized the right things; that our points were clear; that we met our objectives; that we have provided people with tools (games) they feel they can integrate into their own synchronous programs. Next go round we may emphasize even more the need to incorporate games as they relate to content, not just as filler, I also see that we may have given the wrong impression about something: there's a comment in the lower right quadrant about self-paced learning, which we didn't discuss at all and certainly weren't casting aspersions toward. (Heck, I'd rather access the worst self-paced program than most lecture-based 'webinars' any day!)

And what else does it tell us? Well, for those who believe that the online experience suffers due to lack of eye contact and body language, look at this screen again: are people interested and engaged? Do I really need traditional "eye contact" to tell me that?

If you must undertake Level I evaluation, try to find something that will give you more meaningful information than "4.5" ratings with no explanation. And pay attention to the feedback!

What other ideas do you have for evaluating at this level?



Evaluation activity submtted by Michele St. Pierre; adapted from an activity in Pike & Solem's 50 Creative Training Closers (Pfeiffer, 1998).

PowerPoint lesson of the day...

Yesterday I watched someone wrestle to remove a background from a photo, using expensive software with a long learning curve, with lots of time consumed and much frustration. It took me exactly 4 seconds using...PowerPoint.

According to the Office Assistant: "You can create a transparent area in most pictures except in animated GIF. (Make these changes in an animated GIF editing program, and then insert the file again.)...The Set Transparent Color option is available for bitmap pictures that don't already have transparency information. It's also available for some, but not all, clip art." (Most of the art I use comes from clipart.com, and have had good luck with transparencies.)

Are you a technical communicator-type?

Are you a technical communicator? Lots of "application training"? User manuals? "Help" data?

I'm running the Instructional Design Institute for the Society for Technical Communications' annual Technical Communications Summit June 1-4 in Philadelphia. This is a big, fun event great for those who deal in technical communication and information (hint to those of you who do "app training"...)

Karl Kapp and Susan Boyd will be joining me for the ID Institute; we hope to see you there!

Using Game-Show Games to Replace Quizzes

Cathy Moore has been having a discussion about "shoestring" approaches, and among her suggestions are using, 1. photos of real people, and, 2. multiple choice questions recast as games. I give both of these ideas a lot of coverage in my books, but here's a quick example in practice.

Based on the "Hollywood Squares" game, this was developed for a new-hire orientation program both as a way to cover fairly dry content as well as helping staff recognize people they'd need to know to both "get around" and "get along" in the office. (I remember the morning one of our bright-eyed-bushy-tailed new employees encountered our agency head-- then Governor-Elect -- and greeted him with a hearty "Whassup!?")

The real program is proprietary but I dummied up a reasonable facsimile in PowerPoint this morning-- enough for you to get the idea. See E-Learning Solutions on a Shoestring and especially Better than Bullet Points for more ideas on using game-show game approaches to replace traditional methods of quizzing.

What does "LMS" mean to you?

Will Thalhimer has a recent post in which he discusses "LMS" as the "Learning Means Sitting" mentality, while Mark Oehlert once called it a "Lecture Management System".

What does "LMS" mean to you?