Evaluating Learning Management Systems

Selecting a learning management system can be like kissing frogs

Grab that lip gloss and pucker up

I'm writing this blog post as a summary of a webinar I delivered at the invitation of Illumeo, a provider of #eLearning to highly regulated fields. The title of my presentation Kissing Frogs might seem a little sensational, but I stand by my belief that it is an accurate assessment. The selection of a Learning Management System (#LMS) is probably one of the most high-risk challenges that organizations face when it comes to establishing an infrastructure for learning. I often hear this process compared to dating.

Make sure you know who you are marrying because you are going to be with this person for a while.

I think this is an apt way to describe the LMS selection process. Learning and development professionals walk into this process with high hopes that enough time has passed such that today's market offerings are much more advanced and sophisticated than they were in the past. This assumption is valid; LMSs have become much more robust and modern, but they are still not the be-all and end-all when it comes to developing your eLearning infrastructure. They are a part of that infrastructure, not the entirety of it.

Choosing an LMS is just like the procurement process for any other enterprise software system, there is no perfect solution, and you will probably need to supplement your LMS with different tools with specialized functions (e.g., video hosting or knowledge capture systems). The selection process is about figuring out which LMS market offering is the best match for your organization and grounding your discernment of your options in your overall instructional strategy for distance learning. I wish I could tell my clients otherwise, but there is no such thing as a perfect or best LMS, there is merely the best LMS for your organization. You may be asking yourself, "Well what is the best LMS for my organization," and with most questions posed to consultants the real answer is, "Well, it depends."

In this blog post, I will lay out some of the basics I would advise you to consider when selecting an LMS and hopefully I can expose a few of the hidden "gotchas" that could make for a stressful implementation if you don't know about them up front. The real key is identifying and prioritizing the features you know you need and then really digging into your evaluation and making the salespeople you will work with earn that commission.

Who can I trust for unbiased information?

I would like to say you can trust Argyle Analytics, but that's a little on the nose -- not to mention self-serving. Instead, I would recommend you spend some time reading the research published by the Advanced Distributed Learning Initiative (ADL). The ADL is a government-funded, research-oriented, unbiased source of information. While their content can be a bit academic and quite thick, it's valuable. The ADL published a white paper in 2016 that summarizes the current state of the #LMS field. If you are having a hard time sleeping, I highly recommend this riveting read. If you are running short on time, then this blog post might help you tackle a few of the issues inherent in selecting an #LMS.

Download the ADL Whitepaper

LMS vs. LCMS; Tow-may-tow, Tow-mah-tow

I want to clear up one detail right now. When I am talking about LMSs with my clients, I am almost always talking about Learning Content Management Systems. In most cases, the difference is of one between “tow-may-tows” and “tow-mah-tows.” There is a technical difference in that LCMS give you precise control over your learning objects than systems that provide you a web front-end for delivering courseware. I almost never recommend a classic LMS to my clients because they are quickly going the way of the dinosaurs. Unless a specific product has features that trump anything else, I recommend you steer away from the classic LMS. For now, assume that when I say, “LMS,” I am talking about LCMSs.

Choose Your LMS Features Wisely Padawan

Depending on how finely you want to slice things, there can be up to -- and often more -- 55 different features you can consider in an LMS. No single LMS will include all 55 features, and you should note that the more expansive the feature set, the less polished or constrained in functionality those features will be. The ADL has done an excellent job of categorizing and analyzing these features, so it would be redundant to re-write their analysis here. I am going to focus on a couple of key features I feel are essential to take a deep dive on. These are features that often people will make assumptions about how they work because they want them to work a certain way or they can't imagine that they would work any other way. I have a few years experience as a web development project manager, and I can personally attest that software designers and developers have many constraints on their time, attention, and concept of what a feature is and should do. Some of these limitations are very real logistical, and some of them are self-imposed. The simple fact is that you need to dig into these features and stress test your assumptions or hopes.

These are the core features we advise you to dig into

There are many directions I could go with this blog post regarding LMS feature functionality. However, I am going to focus on the following areas.

• Systems integrations
• Notifications management
• Content management
• User-friendly admin interface

I am also going to discuss a few features that never really live up to expectations.

• Reporting
• Knowledge management
• Gamification

I don't believe I am capable of making you an LMS expert by the end of this post, but I will do my best to make help you have more savvy conversations with your LMS vendors. These discussions are the difference between getting an LMS you can live with or shopping for a replacement in three years.

LMS Systems Integrations

LMS Systems integrations are very gnarly things that can make or break your implementation. If you need your LMS to talk to other systems, like your HRIS or your ERP, then you need to be as detailed and specific as you can be when talking to your vendors. Your organization needs to have a Single-Sign-On (SSO) solution for these systems to authenticate their digital bona fides and exchange data securely. Another thing you need to grill your salesperson about is their documentation explaining and detailing their integration. For an integration to work, your LMS and the connecting system needs to have an Application Programming Interface or an "API." The presence an API for both systems is not optional. An API allows your systems to talk to one another and pass data back and forth in predictable and consistent ways. All APIs have limitations; they aren't magic boxes. You need to understand not only what the boundaries are, but what the edges of your other systems are. Clear and easy documentation is essential to a smooth integration because it will allow your LMS administrator and IT department to evaluate if the integration will meet your needs. Without clear documentation of an LMSs API, your quality assurance costs during implementation will be much higher than even your most pessimistic estimates. 

Native integrations are starting to appear in a lot of off-the-shelf options. These integrations typically have a point-and-click interface you can configure with minimal technical skill, but this ease of use comes at the cost of flexibility. Your integrations will be more constrained, but much easier to manage without the technical support. A Salesforce integration is most common for organizations that sell their training for any number of reasons. An LMS integration allows these companies to automate some of the record keeping between the LMS and the CRM. The LMS tracks learning experiences, track learner records, and the CRM tracks sales and enables marketing automation. Fully engage your IT team as a stakeholder and that you have a technical expert who can speak to the API capabilities of your connecting systems, how your organization does SSO, and any of the other advanced integration features you need to satisfy your strategy.

Notifications Management

Most organizations lack coordinated approaches to email and flood employees with it. If employees can get away with ignoring most other notifications, they will surely ignore notifications from the LMS. SMS text messages are one way to get past the “email blindness”; there are some services that issue reminders like this, medical practices being one of them. If you have an LMS with a mobile app, ask your vendor if you can configure push notifications that will deep link your learners to the specific training content that needs their attention. Instant messaging platforms are great ways to cut down on email and still get the message across; Docebo, for example, has a native integration that lets you notify learners via Slack instead of email. 

Do not assume that the system will be easy to use or intuitive when it comes to selecting the triggers and conditions that will drive your notifications; LMS vendors may not invest the time and effort into this function you would assume they would. Don’t let them saddle you with a half-baked feature set. Map out your notifications and detail the conditions and triggers you would need to make them happen. Review your notifications map with your vendors. Above all, do not expect that the perfect system will have the ideal notifications. It's important to keep things balanced, but it never hurts to make an informed decision.

Content Management

Content management is one of the most fiddly bits of any LMS. A lot of organizations struggle with developing a training content strategy. This means they struggle with organizing their content in a manner that makes curating and maintenance difficult and costly. Native content management systems allow you to create a basic level of content with the option of including streaming video from hosting services; then this is a cost-effective solution that doesn't require a lot of skill with courseware development.

Native content also relies much more heavily on how it's organized and represented regarding the user interface. All the organization on the backend doesn't matter if it doesn't look decent on the front-end. You need to welcome feedback on the system and make modifications when they make sense. This trade-off for only basic content means you spend less time managing a navigation experience within courseware and more time on the precision of the instruction in your instructional media. If you have elementary training needs -- such as awareness training or passive information delivery, this is a great way to reduce costs, maintain quality, and track for accountability.

I advise you to stress test it and pull together a quick course using only the native tools. Have your instructional designers create a sample course using just native tools and have one of your “power learners” give you feedback. Discuss that feedback with your vendor and decide if you liked the experience enough to commit to the product.

Having worked on multiple digital implementation projects spanning open source websites, SharePoint content projects, and LMS systems I have the following advice -- don’t fight the system, you will lose every time. Off-the-shelf solutions always require lots of configuration and can be quite flexible, but they all have limits and a design intent behind the administration of your content. The designers of the software have no way of knowing the unique content strategy for your organization and how you organize that content. LMS developers conduct industry research and are usually specialists in this area. These developers will practice what project managers call "expert judgment" which is always code for making educated assumptions. All LMS Developers need to make reasonable assumptions about how training organizations operate in the most general sense.

You need a plan for how you will leverage the baked-in features of the system -- you need a plan for how you will adapt to the system and not the other way around. Content migration is probably one of the most high-risk parts of any implementation. You have content that needs to be staged and set up in the system. Don't forget that you have training records that will require importation -- talk with our vendor and try to negotiate some technical support out of your contract.

Most importantly, you need to plan for errors and issues to emerge -- there is no such thing as a perfect or clean migration, only ones that could have gone far worse. A content migration is a time to focus and eliminate sources of risk. For the most ambitious members of our audience today I have one recommendation -- do not bite off more than you can chew; avoid trying to fold in enhancements or other improvements until after you’ve completed the migration.

“Job Aids” is the common parlance for what training wonks call “performance supports” -- in this day and age there is almost always a digital aspect to this. If you have forms and templates to support on-the-job performance you have to decide if they will live in your LMS or another system. I recommend storing them in another repository; ideally, your intranet where the content is discoverable via a search engine -- make sure you understand how these systems can be hyperlinked (e.g., sharing links from Google Drive, MS OneDrive, your in-house file server, SharePoint, Confluence, etc.). Take the time to think through a multi-channel content strategy for inside your organization; if you already have one, think about how you can leverage that along with your new learning management system.

Administrative User-Interface 

This is where I get to be the bearer of bad news, but I have yet to encounter any LMS admin that says, “I love the admin experience.” In my opinion, this will always be the one area where vendors skimp on investment because they figure that the people buying the systems are not the end-users of the systems. By the time they figure out how subpar the admin experience is, you’re already wedded to the LMS via contracts and labor costs. Dig into this part of your evaluation. Test drive the UX as much as your schedule and budget allow. Set up a course, go through all the steps you currently do and make sure the vendor is there to support you while you test drive it -- make that salesperson earn that commission. Click on everything and see what happens. If your organization requires mobile learning, make sure to also look at the UX in mobile -- most LMS systems will provide minimal to zero features for admins and instructors via the mobile app -- this has implications for your instructors if they are on-call to support learners outside regular business hours.

Moderate your expectations now because the interface will never match your expectations. There are lots of reasons why the admin UX is constrained, but those reasons won’t impact the price you pay. You should not expect or plan that your admin costs will decrease no matter what your vendor tells you. The UX will take some getting used to no matter which LMS you choose -- none of them are going to be intuitive enough that you will “just get it” right off the bat. Look at the training and user documentation very carefully. Many vendors will offer training as part of their customer success program -- if they do not then you should look for another vendor. Don’t just take the “we offer training” sales pitch at face value, use your knowledge of instructional design to evaluate their offerings.

Make sure you inspect your vendors' product documentation. The documentation should include examples, case studies, high-quality YouTube tutorials. Bonus points if the vendor offers a knowledge base that’s indexed by Google and supports questions like, “How do I create notifications in my LMS." Use your instructional design savvy to determine if this is sufficient for your needs. Even after all this review, it’s pretty much a guaranteed that their training offerings and product documentation will come up short on all fronts. Make sure to leverage your service agreement and get your value out of this.

LMS features that never live up to your expectations

My claim that some LMS features are just limited in their sophistication is something of a subjective judgment call on my part. But I feel reasonably confident in my assessment. A feature might be just the right thing for some organizations, for others that feature will be a major let down. In some cases, the features will only be half-baked, in other cases the feature simply will not meet your needs no matter how well executed the feature is. The following features are ones that I think you should consider very carefully since I don’t believe any LMS vendor has gotten it right.

Reporting

Your reporting needs are unique to your organization. Vendors will rarely replicate the logic behind your evaluation efforts, the reporting formats, or the data exports. They need to design the product to have a broad appeal which means it's an evaluation jack of all trades, but master of none. You can expect to get great reports on user activity like registrations, logins, and primary metrics around interaction with content. These metrics are helpful for the direct management and accountability for trainees handing in their homework. 

You can expect to conduct surveys and collect data for Level 1 (Reaction to training) surveys, but don’t expect a fancy data visualization studio. You can expect to get data exports that will require advanced knowledge of Excel or Google Sheets to conduct sophisticated analysis of your tests or Level 2 (knowledge checks) assessments looking for broken test questions. Do not expect any features that will reduce the effort on a Level 3 (Performance improvement) or Level 4 evaluation (Impact). This is because the methodology for evaluating these levels of organizational capacity will be so unique to each organization that you cannot design a computer program around the assumptions that would underlie any evaluations program. Additionally, some of the most sophisticated reporting in learning analytics requires a Learning Record Store fed by a well designed and mapped set of xAPI statements. If you are heading down this road, you need to make sure you discuss this with your vendor.

Knowledge Management

Knowledge management is a tangled and sticky mess for most organizations. Knowledge gets locked in various file formats. These file formats are organized in an inconsistent manner. Much of the real knowledge of an organization survives via a “tribal lore” model that barely scales past a few dozen employees. Shared network drives require people to know where the knowledge is ahead of the moment of need when it’s valuable. Intranets are treated like the old gray mare rather than a well-bred stallion of performance improvement, and I believe that they are worth the investment.

Your LMS is not going to solve the bigger problem rooted in a lack of a strategy for internal communication and stewardship of intellectual property. Here are the facts folks, your learners will almost never return to the training materials for a refresher if you don’t design all your learning as though it were micro-learning. As an example, compare how Lynda.com designs their video course compared to how your organization creates its video learning content. Your LMS will have a search engine, but cannot index content if you lock it in courseware. Your beautiful Articulate or Storyline eLearning courseware is not searchable and is of limited value as an EPSS. Rather than harp on these issues, you should be looking to lay claim to your intranet. Most modern intranet tools are pretty robust -- SharePoint, Confluence, Bloomfire and even Google Sites are all great options, but they require a specialized strategic thinking skill you may not have. Look to content and multi-channel digital marketing for lessons you can apply to curating just in time content for your learners. 

If you can get the content developers on board, and work out a governance process that distributes a portion of content creation back on to SMEs, you can gradually build a culture of knowledge management in your organization. Teach your team how to record better webinars you can slice and dice into micro-learning. Many of you have probably heard the trend that quality expectations are lower for internal-only training content. This is true, think at all the death-by-bullet point webinars you’ve sat through just to hear, “Hey gang! I will post this 90 min. Webinar on the share drive so you can watch it again if you need to” -- Who is going to hunt that video down and scrub through 90 min. of noise just to find the 5 minutes worth of content that applies to your immediate need?

One of my clients was doing just that. We started working with the SMEs on improving their presentation skills so that we could slice and dice their webinar into small 5 minute chunks which we embed from a centralized video host throughout their intranet along with the summary text. We are also working on some lightweight SEO best practices for content development. This gives employees a more Google and YouTube-like experience when searching for information. It requires a bit more work on the front end, but the costs associated with the intentional curation of video collateral is less expensive than the process loss and wasted time of dozens of employees search for information and interrupting their coworkers.

Get a video hosting service. The same client I mentioned regarding webinars also signed up for a video hosting service. Any LMS worth its codebase will support streaming from video services like YouTube and Vimeo. My client is using Vimeo because it makes a streaming video to multiple sources painless and gives you a centralized platform for managing all video collateral that supports your training program. Learn how the search function on your intranet works under the hood and start search engine optimizing your organization's knowledge base. Search engines are not magical; they work by reading web pages and indexing the content so that when you type keywords or search terms into Google, you get the results you want. Content marketers and bloggers learn very fast how to use keywords and search terms they know their audience uses when searching for information. Good search engines need good content, and good content boils down these two things -- good solid writing and using pre-programmed styles for sections headers on web pages. If your organization can develop this basic competency, then you can create a more Google-like experience for your learners.

Gamification

When I was in grad school, I remember reading a few articles and attending a conference session that featured analysis of World of Warcraft and Second Life. What a nifty example of digital collaboration! What a fantastic and fun way to teach people! Learning experts should take heed of these technologies because this is the future of learning! At the time I was an avid World of Warcraft player -- I rocked the house on heals with my resto speced Orc shammy -- and remembered thinking, “This is what not getting it looks like.” Second life is all but defunct and never took off as a virtual learning environment, and World of Warcraft is complicated, but when you play it four hours a day for three years you pretty much learn everything there is to know about a tremendous waste of time. 

Gamification does not mean turning your learning experiences into video games; most of the examples I’ve seen that I would consider on par for the industry use game mechanics that are a bit childish and more appropriate for learning games aimed at grade school children; this can come across as condescending and burdensome to learners. Gamification is really about reward, risk, and working the system to achieve the goals of the game. You can gamify learning, but it has to connect to something tangible like a promotion or a raise. Most LMS that offer gamification primarily boil it down to earning badges. Unless your organization has a broader strategy for assigning a tangible value to earning a badge, then don’t bother factoring it into your decision making. 

The hot new thing I hear in the gaming space is Augmented Reality and Virtual Reality. You might be thinking about Google Glass, but you should know that Google discontinued that product so you should keep that in mind the next time you hear someone talking about Google Glass in the presentence. Microsoft Halo is a long way off from being a mature market offering; content is expensive, and creating virtual learning environments is more appropriate for blue collar jobs with dangerous working conditions. I think these systems would be brilliant for training firefighters on the dangers of a burning building, but awful for training people on Microsoft 365. While I do believe there is promise for AR as a form of performance support -- think of a Xerox tech making a house call or an auto technician working on an exotic engine -- you're off the shelf LMS is not likely to support AR or VR content anytime soon.

The workforce will skew younger as the Baby Boom generation transitions out of the workforce you have to consider that the up and coming generation was raised in a game-rich environment. This group is very game-savvy and has a well-developed mental-model of what constitutes a high-quality game and what seems cheesy and hoaky. With a few notable exceptions most of the “training games” I’ve encountered come across as childish and hoaky. The best one I’ve seen recently is a board game for how to run a hotel developed as part of a management training program at a major hotel chain. In the case of a board game, just make sure your LMS can support the tracking of instructor led training courses in addition to the management of eLearning courses.

Unless you are going to make an effort to be best-in-class and fight the pressure to cut corners, just don’t do it.

In Conclusion

I'd like to close by saying that even though this post reads like a cautionary tale, there are some excellent off-the-shelf LMS. There are also some interesting examples of LMSs built on #WordPress and #Drupal. I've worked with both systems, and I must say that they offer an interesting business opportunity to anyone who has the rights to sell access to a library of valuable content.

Thank you for taking the time to read a rather long blog post. In exchange for the gift of your time, I would like to offer a gift of my own. Please take a look at my Google Sheet-based template for a vendor scorecard. Included, is a set up that will help you tabulate scores for LMS vendors as you weigh your options. I've embedded some notes to help you along, so look for them as you review the template. If you have any questions about the template, please reach out, I would love to hear from you.

Download Our Template

This article was first posted on argyleanalytics.net

Related Hashtags: #LMS #training #elearning #wordpress #drupal