Sep 09 2013

How do we evaluate the strategic use of collaboration platforms?

Published by at 3:23 pm under collaboration,evaluation

The earthHey smart people, especially my KM and collaboration peeps, I need your help!

I’ve been trolling around to find examples of monitoring and assessment rubrics to evaluate how well a collaboration platform is actually working. In other words, are the intended strategic activities and goals fulfilled? Are people using it for unintended purposes? What are the adoption and use patterns? How do you assess the need for tweaks, changed or deleted functionality?

I can find piles of white papers and reports on how to pick a platform in terms of vendors and features. Vendors seem to produce them in droves. I certainly can fall back on the Digital Habitats materials in that area as well.

But come on, why are there few things that help us understand if our existing platforms and tool configurations are or are not working?

Here are some of my burning questions. Pointers and answers DEEPLY appreciated. And if you are super passionate about this, ask me directly about the action research some of us are embarking upon (nancyw at fullcirc dot com)!

  • How do you do evaluate the strategic use of your collaboration platform(s) and tools in your organization?
  • What indicators are you looking for? (There can be a lot, so my assumption is we are looking for ones that really get to the strategic sweet spot)
  • Does the assessment need to be totally context specific, or are there shared patterns for similar organizations or domains?
  • How often do you do it?
  • How do we involve users in assessments?
  • How have the results prompted changes (or not and if not, why not)?

Please, share this widely!

THANKS!

15 responses so far

15 Responses to “How do we evaluate the strategic use of collaboration platforms?”

  1. Harold Jarcheon 09 Sep 2013 at 3:45 pm

    This is mostly from a technical perspective, but may be helpful (PDF):
    http://www.jarche.com/wp-content/uploads/2013/09/IT-Selection-Lens.pdf

  2. Nancy Whiteon 09 Sep 2013 at 3:51 pm

    Thanks, Harold. Do you know of anyone who has used these lenses to actually do an assessment of use? Have yoU? What I’m finding is I can find quite a few theoretical lenses, lots of potential indicators, but not alot about how to actually use them to measure. (Now, back to reading your PDF. Thanks again!)

  3. Dave Pollardon 09 Sep 2013 at 8:31 pm

    I confess, after years as a so-called ‘CKO’, that I am deeply suspicious of most of the measures of ‘success’, ‘progress’ etc. in complex systems. As curmudgeon Snowden says, there’s a propensity in humans to cognitive biases that see patterns where there are none, find convergence and consensus where there is none, and see causality between actions and results where there is none.

    My sense is that the best way to evaluate any program, tool, platform or process is to set aside specific time for assessment, have an open and candid conversation about it, and ensure the rewards for ‘success’ don’t bias us too much in our assessment. An appreciative approach I think works best i.e. what could we do to make this better, what could we do instead, what could we stop doing without major consequences to free up time for other work etc.

  4. Nancy Whiteon 09 Sep 2013 at 9:22 pm

    Dave, so far this approach sounds the most rational from what I’ve seen. What I hope to explore is how to frame such conversations. I notice a lot of a) lack of self awareness of how people use a platform (which isn’t a bad thing itself, but makes reflection a bit tough) and b) often very little clarity of “why bother.” And we wonder why we have challenges with “adoption.”

    I have also been toying with an idea w/ Rachel Cardone to look at activities and then see if there is some sort of useful view from the Cynefin perspective. So many possibilities! Thanks

  5. Dave Pollardon 09 Sep 2013 at 9:32 pm

    Might be some synergies here with what Christopher Allen, Tim Bonnemann et al are thinking about w/r/t facilitation patterns in virtual environments/meetings. On the recent call someone said (to many nods of agreement) they just HATE virtual meetings, period. They just find them intrinsically dysfunctional compared to F2F, and have despaired that any kind of non-F2F collaboration/decision-making can ever be particularly effective. That’s a deep hole to climb out of, even if it’s a false perception.

    Agree completely on your two ‘noticings’. When we’re largely unaware of how we work as individuals, it’s pretty hard to become aware and helpful w/r/t work with others.

  6. Maureen Crawfordon 09 Sep 2013 at 9:35 pm

    I suggest looking at Maria Mayan’s book, Essentials of Qualitative Inquiry.

  7. Nancy Whiteon 10 Sep 2013 at 6:48 am

    Thanks, Maureen! I’ll go fetch the book right away!

  8. Michelle Laurieon 09 Sep 2013 at 11:32 pm

    Hi Nancy,
    Very interesting and sounds like fun đŸ™‚

    You may find something of use in this paper on starting points for assessing collaboration: http://www.iisd.org/publications/pub.aspx?pno=1741
    You may already know the answers but some initial questions to help focus (that will change the types of questions you ask) would include:
    – is the platform internal to an org or external/cross-organization/networked?
    – who is the evaluation for?
    I do think that Cynefin could be used as another way to organize the framework. I imagine that some things on the platform should be simple and straightforward like uploading files, searching for people while others could be more complicated such as discussion forums and perhaps even complex in terms of what emerges from the platform. Once that is delineated you could organize the review/questions to ask accordingly.

    Good luck and keep us posted.

    Cheers,
    Michelle

  9. Nancy Whiteon 10 Sep 2013 at 6:51 am

    Thanks, I have that one on the reading list already! Kismet! And we have at least 5 people evaluating platforms in their org, so we have internal, cross, and external. We want to learn across as well as within a single type of use.

  10. Tim Bonnemannon 10 Sep 2013 at 4:59 am

    Just saying hi so I can subscribe to new comments on this thread. đŸ˜‰

    And yes, really hope you’ll be able to join one of our upcoming “online patterns” calls/meetings, Nancy!

  11. Nancy Whiteon 10 Sep 2013 at 6:48 am

    Hiya Tim. Do you know Aldo deMoor? I’m working with his collaboration patterns. Dave, do you know him?

    Re virtual meetings… it is interesting. I’m starting to find people who are doing them well, but they are small in group size. The ones that seem to have worked with larger (not HUGE… i.e. 50 or less) seem to work when the meeting purpose itself was transformative and the chat was powerful. So many of them are awful. But so many F2F meetings are awful.

  12. Intranet Loungeon 10 Sep 2013 at 12:14 pm

    How do we evaluate the strategic use of collaboration platforms?

    This article has been submitted to Intranet Lounge – Trackback from Intranet Lounge

  13. […] I think this is an excellent idea and, alas, for me, the timing sucks. I have put a red line on my to do list promising myself not to bite of anything else. (I’m doing an action research project on how to usefully evaluate the socio-technical stuff associated with organizations’ “collaboration platforms!!” And yes, i bit off WAY more than I should have! See this previous post) […]

  14. Catherine Borgman-Arboledaon 18 Sep 2013 at 1:52 pm

    Hi Nancy,
    I did this report for UNDP Vietnam a few years ago, focused on effective practices of KS and org learning for umbrella NGOs. There are a few sections where I looked specifically at collaboration platforms, infrastructure and technology which you might find useful. If I recall correctly, one of the clearest success indicators was sustainable use over time, and obviously evidence that people were actually better equipped to do their job/carry out a specific task, through use of a platform. You are probably looking for more granular indicators, but this research came to mind when I read your post — (report is here) https://docs.google.com/file/d/0B1bwEHfQTeaiSGpDbVcwaFRwWnc/edit?usp=sharing
    Best,
    Catherine

  15. Nancy Whiteon 18 Sep 2013 at 4:09 pm

    Thanks, Catherine, I really appreciate the link. I’m hopping to it now!

Creative Commons Attribution-NonCommercial-ShareAlike 3.0 United States
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 United States.
%d bloggers like this: