Learning While Building eLearning: Part 2 – Learning Objectives & Assessment

signsThis is the second (slow!) of four pieces reflecting on the experiences of Emilio, a subject matter expert who was tasked with converting his successful F2F training into an elearning offering. Emilio has let me interview him during the process.

This piece focuses on the thorny issue of learning objectives at the front end of an elearning project and assessment at the other end. You can find the context in part 1 here. (Disclaimer: I was an adviser to the project and my condition of participation was the ability to do this series of blog posts, because there is really useful knowledge to share, both within the colleague’s organization and more widely. So I said I’d add the blog reflections – without pay – if I could share them.)

Nancy: Looking back, let’s talk about learning objectives. You started with all of your F2F material, then had to hone it down for online. You received feedback from the implementation team along the way. What lessons came out of that process? How do we get content more precise when you have fewer options to assess learner needs and interests “in the moment” as you do face to face, and with the limited attention span for online?

Emilio: I realize now that I had not thought about being disciplined with learning objectives. I had created them with care when I first developed my F2F offering. Once I had tested the course several times, I recognized that I forgot my own initial learning objectives because in a F2F setting I adapted to student’s interest and knowledge gaps on the spot, and I was also able to clarify any doubts about the content. Therefore, over time, these learning objectives become malleable depending on the group of students, and thus lost presence in my mind.

This became apparent as  I was doing the quizzes for the online work and got comments back from Cheryl (the lead consultant).  She noted where my quiz questions were and were NOT clarified in the learning objectives and content. I realized I was asking a bunch of questions that were not crucial to the learning objectives.

With that feedback, I narrowed down the most important questions to achieve and measure the learning objectives. It was an aha moment. This is something that is not necessarily obvious or easy. You have to put your mind into it when you are developing an e-learning course especially. It applies to the F2F context as well, but in an e-learning setup you are forced to be more careful because you cannot clarify things on the spot. There is less opportunity for that online.  That was very critical. (Note: most of the course was asynchronous. There were weekly “office hours” where clarifications happened. Those learners who participated in the office hours had higher completion rates as well.)

It was clear I had to simplify the content for elearning set up – and that was super useful. While my F2F materials were expansive to enable me to adapt to local context, that became overload online.

Nancy: What was your impression of the learners’ experiences?

Emilio: It was hard to really tell because online we were  dealing with a whole different context. Your indicators change drastically. When I’m in F2F I can probe and sense if the learners are understanding the material. It is harder online to get the interim feedback and know how people are doing. For the final assessment,  we relied on a final exam with an essay question. The exam was very helpful in assessing the learner’s experience, but since it is taken at the end of the course, there are no corrective measures one can take.

Nancy: Yes, I remember talking about that as we reviewed pageviews and the unit quizzes during the course. The data gives you some insight, but it isn’t always clear how to interpret it. I was glad you were able to get some feedback from the learners during your open “office hours.”

We used the learning objectives as the basis for some learner assessment (non graded quizzes for each unit and a graded final exam which drew from the quizzes.) How did the results compare with your expectations of the learners’ acquisition of knowledge and insights? How well did we hit the objectives?

Emilio: We had 17 registered learners and 7 completed. That may sound disappointing. Before we started, I  asked you about participation rates and you warned me that they might be low and that is why I am not crying. The 7 that completed scored really well in the final exam and you could see their engagement. They went through material, did quizzes and participated in the Office Hours. One guy got 100% in all of the quizzes, and then 97% in the exam.

We had 8 people take the final exam. One learner failed to pass the 70% required benchmark, but going deep into it, Terri (one of our consultants) discovered the way Moodle was correcting the answers on the multiple choice was not programmed precisely. It was giving correct answers for partially correct answers. We need to fix that.  Still, only one failed to pass the 70% benchmark even with the error.

The essay we included in the exam had really good responses. It achieved my objective to get an in depth look at the context the learners  were coming from. Most of them described an institutional context. Then they noted what they thought was most promising from all the modules,  what was most applicable or relevant to their work. There were very diverse answers but I saw some trends that were useful. However, it would be useful to have know more of this before and during the course.

Nancy: How difficult was it to grade the essays? This is something people often wonder about online…

Emilio: I did not find it complicated, although there is always some degree of subjectivity. The basic criteria I used was to value their focus on the question asked, and the application of all possible principles taught during the course that relate to the context described in the question.

Nancy: One of the tricky things online is meaningful learner participation. How did the assessment reflect participation in the course?

Emilio: We decided not to give credit for participation in activities because we were not fully confident of how appropriately we had designed such activities for an e-learning environment in this first beta test. I think this decision was the right one.

First, I feel that I did not do a good job at creating an atmosphere, this sense of community, that would encourage participation. Even though I responded to every single comment that got posted, I don’t really feel that people responded that much in some of the exercises. So I would have penalized students for something that is not their fault.

Second, we had one learner who did every exercise but did not comment on any of the posts. He is a very good student and I would have penalized him if completion relied on participation. Another learner who failed did participate, went to the office hours and still did not pass the final exam.

We failed miserably with the group exercise for the second module.  I now realize the group exercise requires a lot of work to build the community beforehand.  I sense this is an art. You told me that it is completely doable in the elearning atmosphere, but after going through the experience I really feel challenged to make it work. Not only with respect to time, but how do you create that sense of community? I feel I don’t have a guaranteed method for it to work. It is an art to charm people in. I may or may not have it!

Nancy: The challenges of being very clear, what content you want to share with learners, how you share it, and how you assess it should not be underestimated. So often people think it is easy: here is the content! Learning design in general  is far more than content and learning design online can be trickier because of your distance from your learners – and not just geographic distance, but the social distance where there is less time and space for the very important relational aspects of learning.

Up Next: Facilitating Online

Relate and Liberate: Techniques for Online Learner Engagment

threelegsOne of the events from Novembers ISS Fellowship in Victoria State, Australia, was at Melbourne Polytechnic. We had a handful of people interested in participating virtually, but I really wanted to give them my FULL attention, not split between the F2F and the online group. So earlier this month we did an hour together online. You can watch the recording here – please note, there is a LOT going on in the chat, so read it if you decide to look at the recording.

The slides and notes artifact is here: RelateandliberateOnlineVersionFinal

So You Want to Host a Web Meeting? A Resource

webconferencingA long time ago in a planet far far away, a group of people asked if I could share some of my web meeting tips. I have a lot of tips, most of them learned from many many colleagues from all over, both from watching the masters work and from resources they have created. Finally, I got around to starting the project. It was supposed to be a “tip sheet” of 1 page, both sides. hahahahahaha…

Because I love my smart friends like Pete Cranston (the instigator, I might add) http://uk.linkedin.com/in/petercranston, Susan Stewart http://guidedmeetings.com/ and Bonnie Koenig http://www.goinginternational.com/about/, I started a google doc. They added ideas, and I started writing.  You can see the genesis here.

Many pages later we have  So Yo Want to Host a Web Meeting? I hope you find it useful, and as always, I welcome comments, suggestions for improvements, additional resources, and catching me if I did not attribute properly. The latter was very difficult because so much of this has been learned along the way and ingrained into my practice. The challenges of standing on so many shoulders!!

Edit: 2/17/16 A great pre-webinar activity for when people are logged on and waiting for the meeting to begin from Rachel Smith at The Grovehttp://www.grove.com/pdfs/Do-Nows.pdf 

Relationship Centric Teaching – Part 3 of ISS Fellowship

This is the third in a series of posts about my ISS/Chisholm Fellowship in Victoria State, Australia. You can find the previous posts here: Part 1, Part 2.

learningLiberationBoth of my weeks in Victoria revolved around a series of workshops that were generally designed around the idea of increasing learner engagement. We played with all kinds of titles in advance, but of course, once I showed up and started to hear people’s stories, the new theme emerged: Relate and Liberate. I was very inspired by this quote:

“If you have come here to help me, you are wasting your time. But if you have come because your liberation is bound up with mine, then let us work together.” Lilla Watson (Quote found via The Interaction Institute/ )

Coincidentally, an essay by Clay Shirky, The Digital Revolution Has Already Happened” was circulating when I was planning and it really hit home. In it Clay talks about the importance the access online learning has provided.

I also wanted to focus on relationship centric teaching using conversational approaches. This was supported by a graphic facilitation workshop, which in the end, applied the relationship centric approach while introducing the joy of visuals and graphic facilitation in teaching and learning. You can read more about that workshop in Part 4.

threelegsFinally, I wanted to try out some thinking that I’ve been doing around how to shift such a strong emphasis on content to a “three legged stool” approach that looks at the interplay between content, relationship and social scaffolding, and signals (quantitative and qualitative data that helps us make sense of what is happening) not just from our courses, but across courses and options made possible by open learning. That will have to wait for a full blog post, but I’ll slide in my sketch here and leave it at that for now.

I was surprised that most of the participants were primarily teaching face to face. In my past visits to Australia to hang out with my educator friends, the emphasis had been much more strongly positioned on the online. So I made sure to talk about both online and offline contexts around the materials and processes. The first group at Chisholm were the Learning Leaders working on community based education. The introduction was strongly tilted towards seeing learning as liberation. I have a deep fondness for community based learning. The subsequent sessions were mostly TAFE educators or designers of learning courses and materials.

Process

In all of the workshops I tried to hold myself to the standard of walking my own talk. My plan was to focus on identity and relationship as a key to engaged teaching and learning, and use methods from Liberating Structures as a set of exemplar processes to embody this approach. That meant a focus on liberating the intelligence and passion in the room, making time for connections and creating conditions for useful conversations. My role was to be a catalyst, rather than positioning myself as the expert. This is a good thing, because I’m a learner first, expert… well, that is way down my “identity” list!

Liberating Structures were part of every workshop. We used Impromptu Networking to identify shared challenges, 1-2-4-All to make sense across those challenges. Then the subsequent structures varied by workshop. We  very successfully used Troika Consulting (I keep calling it by the name I know – Triad Consulting!) and Discovery and Action Dialog (DAD) to help address the challenges each group identified, W3 to evaluate the session, tagging on 15% Solution as the “What Next” step of W3 to identify a simple follow up step. In some of the workshops we ended with a simple appreciative networking activity to note who contributed to our experience during the workshop, and who people wanted to follow up with.

goatrodeoIn each of the workshops I offered a quick overview of Liberating Structures (see slides) that covered the micro structure concept and some other example structures. But I have found it has been more useful to USE them, then as appropriate, debrief them, rather than “preach” them.  I reviewed the basics of LS by showing a slide about the micro structures, the list of the 33 structures and shared Keith McCandless’ recent thinking about that (fragile) and rich space between over control and under control (goat rodeo – see Keith’s image to the right!) In the workshops there was insufficient time to talk about how to build an entire agenda by “stringing” structures, so I have included some examples at the end of the slide deck. That probably should have a blog post of it’s own!

In the session where we did DAD, I really appreciate the reflections about the value of iteration in DaD, and in staying close to the questions that are at the core of the structure to avoid “goat rodeo.”  Goat rodeo is everyone doing their own thing. Smart people fall into this trap all the time. In Troika, many people mentioned the freedom of turning one’s back to listen in. In all the structures people noted the deep importance of the starting questions. The more specific the question, the more precise answers are liberated.

A fabulous question was “when is it appropriate to use LS.” I offered an answer, but I also suggested I email everyone in a month and find out what they have used and done, and we’ll generate an “in situ” answer — nothing like reality!

Participant Feedback

In most of the workshops we did the “What, So What, Now What” debrief and reflect Liberating Structures. I was able to capture a few responses on video. Here is the result:

In addition, I received this quote in the mail this past week with permission to quote anonymously:

Hi Nancy,

Just a quick not to say thankyou for coming out to Australia, visiting us and giving us insight to your perspectives. 

 Can I just say that I thoroughly enjoyed it and put a couple of things into practice, nothing special but I went into class with a much more open mind and content within myself.

 I ended up combining 2 groups from 2 campuses for the final 5 classes and although the first night’s turnout was a little lacking, by the end they were developing new workgroups, mixing on their own, helping each other in understanding assessments and to top it off they even arranged a Christmas breakup for both groups together.

 The last night was purely a submitting work  and as a help session for those that hadn’t finished or submitted all their work and I still had nearly a full class!  They brought in cakes, all shook hands and celebrated and it was a genuinely nice thing to see.  Especially when most stayed around until 8pm on a work night.

 I wish you all the best in your travels and business.

Take care,

My Reflections

IMG_20151118_154110941Identity & Good Teaching

This issue came up most strongly in the workshops hosted by eWorks the last day of my fellowship. I took this little visual note on the white board. Our conversation about educators having a strong self identity as educators was the basis of good teaching. Good teaching comes before any facility with online teaching. It always goes back to those basics. This is no surprise, but surprisingly this concept can get lost with online initiatives because people focus so intently on content. Content alone can be found many places. The unique offering of the TAFE institutions is GOOD TEACHING.

Conversational Teaching

IMG_20151112_141319178An essential practice of good teaching – online or offline – is getting immediately into good and useful conversations. I asked people at many of the workshops if they struggled with discussion boards and many raised their hands. I suggested that we need to think carefully and skillfully about how we engage people so that things like discussion forums and web meetings are meaningful, not just things learners have to to. NO TICK THE BOX! This is where we can always improve our skill at designing really engaging questions that people can’t resist responding to, versus canned “discussion prompts.”

In our workshops, every session was started with a conversational approach that asked people what they wanted to get out of the session and what they had to offer. This activity helped me know what they wanted, and acknowledged their expertise as educators and designers of learning. The process used rotating paired conversation and without fail, the buzz in the room was robust and it was always hard to get people to stop talking. I take that as a sign of engagement! (Yes, they could have been complaining about me or the process… 😉 ) But again, this acknowledges identity in the context of meaningful conversation.

I asked people how they currently open conversations in their teaching, and how they might change this. One person said he was going to take is face to face group to coffee, instead of starting by reviewing the syllabus. Another was going to use the paired drawing exercise we did in the graphic facilitation workshop to help learners create relationships right from the start. Just two examples!

It was interesting to be in rooms with so many smart and passionate people, yet I sensed a reluctance for people to speak up at the full group level. Is this part of the identity thing? IS there a “tall poppy syndrome” issue in these organizations?  It may be some of those things, but for me it was yet another example of the critical importance of breaking people into smaller groups because intense, buzzing, engaged conversation emerged every time at the small group level.

brainBrain Based Approaches

Before the workshops I happened on a fascinating article on neurobiology. It described how neurobiology might inform our teaching practices, particularly the work of Dan Siegel. He talks about the unity of the “triume brain” of cerebral cortex (rational brain), the limbic system (emotional brain) and the stem (reptilian brain). Siegel “envisions the brain as a social organ,” and “the emotional system that develops in relationship.” I was taken how he describes a “sixth sense” as “mindsight,” and links this to mirror neurons. “What fires together, wires together,” is a way he talks about how we learn by what we observe. If we observer our teachers functioning as learners, will we be better learners? I think so…  Siegel talks about the power of associations that people make in order to make sense of the world. Positive and uplifting associations can be more meaningful, encouraging, and benefit change. There was so much in this and I only scratched the surface. But by the second week I had to make a visual…

Remember Group Process

A post on Facebook by the fabulous Chris Corrigan reminded me of some very resonant practices from the Art of Hosting and I grFrom Chris Corriganabbed an image to share about the Four-Fold path of Presence, Contribution, Participation and Co-Creation. I am a little shocked when I don’t see many of the deep process work from the facilitation community in teaching and learning. There are natural connections. So introducing across these communities is a particular joy. Going by to my “three legged stool” — this is the relational aspect. How we interact is as important as what we are interacting about.

Share Real Examples

Finally, it was fabulous to hear the examples of the educators in each workshop. In turn, I was able to share about a project I’m working on with an international team sponsored by the Justice Institute of British Columbia and the University of Guadalajara, the  UdG Agora Project. You can take a peek at a recent presentation online about the project from #OpenEd15.

Slides & Resources

A Webinar on Facilitating with Technology With 36 Co-Presenters

NancyAtWorkOver the past few years I have enjoyed being part of the MAFN Webinars. MAFN is the Mid Atlantic Facilitator’s Network. They offer both online and offline professional development gatherings. Last Friday, Amy Lenzo, Nancy Settle-Murphy and I were ostensibly the “presenters.” I think I want to shed that title. May I be done with presenting, please? And by the end, I realized it was a conversation with 36… so they are all presenters too!

We designed the 90 minutes to minimize the presentation and maximize the engagement. I appreciated both Amy, Nancy and the great MAFN team of Michael, Dan, Fran and Meaghan — all ready to try new things and stretch ourselves.

We each had 5 minutes, then we were, as they say, off to the races. (You can see the five questions here in the PDF –> The Edge o fOnline Facilitation MAFN March2015 4 Blog.) In this case, it was a chat race. (At least it felt that way to some folks. I love it. Maybe I was a chat racehorse in my past life?) We then finished this first section with the following question:

HighestAspirationforOnlineEngagement

There were 36 participants (maybe a few less with a couple of duplicate log ins, etc.) and there was this amazing chat stream of ideas. So fertile and interesting. I wanted to share the key points.

  • How can we network our intelligence to solve the huge problems facing the world/us?
  • For on-line engagement to be as powerful and impacting as any face-to-face activity.
  • Better connect with people all over the world.
  • . . that it might be superior in some respects to face-to-face engagement
  • People feel like they are in the same room even though they are notaspirations
  • Making deeper (non-trivial) connections with people at a distance
  • For participants to feel energized and that they have had an experience
  • To “level-set” the stakeholders’ understanding of the problem or challenge that all are facing.
  • When the offline culture is not very participatory – using online/virtual to create exchange and relationships and change the culture a little bit.
  • I think work has to meet human social needs.  Online engagement is quite powerful because it can bring  people together socially around quite microspecific topics and concerns
  • People feel a sense of community, leave with  decisions and catalyze action
  • To introduce & discuss the change process in a positive energising atmosphere when some people are resistant to change
  • Participants learn, understand, and feel connected to one another
  • Superior to face-to-face in some ways because many people can “talk at once”.  Appeals to some participants who might be hesitant to speak in a standard classroom
  • Downplay power distances
  • People actually “hearing” the words others were saying; a participant once exclaimed: We were using the same words but we did NOT mean the same thing
  • People who are not technically minded can participate easily and are engaged
  • Human connection across geography with powerful problem-solving
  • Doing strategic planning for a global association on-line (different time zones, cultures, etc., in addition to all the normal human differences)
  • Online engagement feels like in-person engagement
  • Environments that are truly connected – where each of us feel deeply connected to ourselves – our own thoughts and bodies and full selves; to each other; to the natural environment and to the larger world we’re part of. Intimacy and Scale.
  • That it truly engage and lead to the desired outcomes
  • The technology is secondary – the community and results are foremost
  • Create a level playing field so all can contribute to the best of their ability
  • Speed and access to expertise
  • Every voice contributes
  • Making on-line meeting not just a one-off  — so make it a practice that folks will get comfortable with over time (it won’t happen the first time, magically)
  • The fact that the presenters can’t “keep u” means this is more participation than would be possible in person
  • The challenge with online is that the human connection is mediated and distorted by technology. My aspiration is that the technology would feel invisible, or better yet, would be a catalyst for connection.
  • Highest aspiration:  hold onto the social aspect of learning
  • My highest aspiration is connection. Task completion is secondary.
  • Shared meaning … Having folks all proclaim: “I see, hear, & feel and understand.”

These 30+ people aren’t thinking small and I was encouraged and delighted. And from there, the conversation proceeded, mostly in chat and the rest of us with mic access sprinting to keep up became a cauldron of ideas and insights. The group segued from aspirations to questions and how to’s. Within the ideas there were also the meta comments about being challenged being in a text only environment, the pace, the sense of both richness and chaos. It is interesting to read back through the chat and try and pick out some key threads, and to discern where people are “coming” from with their insights. Some are clearly self-defined as trainers, others as facilitators. Some carry the context of the type of organization they work for, and others as consultants range across contexts.

Here are a few examples:

  • What we notice and aspire too is obviously informed by WHAT we do. There were trainers in the group. People embedded in and in the context of organizations and their constraints. Consultants who ranged across contexts. This informs the type of “how-to” people sought or suggested.
  • Everyone is interested in technology, but often in very different ways. Some want to know about the latest technologies. Others are interested in the impact of technology on our interactions. This quote really drew me in after someone talked about the way technology can distort our experiences: ” The presence of technology DOES change and effect the HUMAN experience and connection.” This segued into a conversation about how technology can/might not level the playing field. We are just beginning to really dig into these issues, and move beyond thinking of technology as a tool. I confess I’m getting tired of “what is the best tool for X” conversations and am ready for deeper explorations into the impacts of our technological environments, regardless of what tool is/isn’t available/acceptable/affordable! (Not to diminish those issues.) This is a great topper to this idea that one of my 36 co-presenters shared: “The way the hammer shapes the hand” — Jackson Browne, Casino Nation.
  • We all struggle. Priceless: “My 20th century mind has been struggling to make meaning and order from the chaos of this group chat.” My observation? It is not related so much to age, but I don’t have data to support my hypothesis (says this 56 year old.) The issue for me from the stress of volume is what is lost, as one person wrote, without reflection. (Slow down, Nancy, slow down!) Clearly I’m not the only one with this issue: “I have not been building in enough reflection time in my webinars.  I think it’s because I can’t tell when people have lost interest or when they are thinking.  Could be an insecurity issue for me.  Need to work on that!”

There was a lot more and I’m attaching a file with my semi-Sorted Chat Notes.

But here is the capper:

CAN WE DO THIS AGAIN NEXT WEEK PLEASE AND TALK ABOUT THE NEW QUESTIONS?

I love my 36 co-presenters! Co-creators! Co-labborators! Thank you, one and all. You help me…

thinkmoreradically