I have spent the past ten years working with Blackboard as a system administrator at multiple higher education institutions in the US. In 2009, Blackboard presented me the Blackboard Catalyst Award for my work within the Blackboard Community. In 2013, I authored the first technical book for Blackboard Learn Administrators published by Packt Publishing.
Currently, I'm the Blackboard Learn Administrator at the University of Missouri. I work with great people and here is the place I post my musings, issues, and finding as I work through the sometimes frustrating, humorous, but always interesting world of Blackboard.
Greetings everyone! Sorry that I haven’t been able to post but I have recently been dealing with a Blackboard production environment that has spent much of its time in various rings of hell. This has happened on and off since the start of the semester (about 14 weeks ago as of the writing of this post). This has been a very difficult time, but there has been one ray of sunshine for us, our implementation of Kibana, ElasticSearch, and Logstash proved invaluable during troubleshooting.
In my opinion, these three applications are the holy trinity of logging for any application like Blackboard Learn. They allow an admin to unlock the information kept in logs at a cost that most IT budgets can handle (the three tools are open source). I learned about these tools from our central server group (CSG) who had heard about the tools during PuppetCon. (See the video below)
They had started testing them in their “skunkworks” area. During a meeting, I lamented the difficulty searching log files on six different application servers. These difficulties made me create the log collection script which I blogged about earlier this year. One of the team members suggested Logstash, ElasticSearch, and Kibana.
The next week I was at my server administrator training at Blackboard HQ in Washington, DC. During that week, one of the attendees from SUNY mentioned these tools as well. These two discussions had sold me. Back on campus, I quickly moved to test out how helpful and easy it would be to implement these applications within our Blackboard Learn environments.
The three tools Kibana, ElasticSearch, and Logstash all have different parts that they play in the log puzzle.
Logstash – This application brings the logs and event data from one or many systems to one location. The open source product runs on top of ElasticSearch.
ElasticSearch – This application is the engine of Logstash. It collects, indexes, parses, and searches the log data.
Kibana – The frontend of Logstash and ElasticSearch, this application is the Google search for all your logs. The user can be as basic or as advanced in how they want to search to find log information. The latest release of Kibana adds more visuals based on collected data.
We were already using shared network storage to hold all the log files from our production, quality assurance, and development instances. This meant that 95 percent of our files wouldn’t be difficult to collect. We could simply mount the shared storage to collect the data. We then had to use a shipper for the other 5 percent that don’t sit with the network storage, such as tomcat logs. The shipper runs on the application server and sends all data written to that file for indexing and storage. We spent a few weeks making sure that the shipper didn’t put any additional load on the application servers. Next we had to learn how to grok.
Grok is a pattern set used by the Logstash application to breakdown logging input and make it searchable. (See Grok Patterns) We created multiple fields for items such as email subject lines, IP addresses, etc. that appeared in the logs. The fields gave us the ability to leverage the data collected and the application’s indexing abilities.
Once all the data was being collected, groked (or parsed), and indexed. The Kibana application stepped into the spotlight. While Kibana has a simplistic interface, it is powerful and sometimes cryptic. I had to create a Kibana Cheat Sheet for myself and my team on how to use the web application to find logging data.
Once implemented, a few team members started to use the information to collect data about our environment. However our fall semester started out with our environment crashing on a regular basis, nearly as well scheduled as a subway train schedule. The only humor was that we would regularly be able to call the time for these failures. “Will the 11:03 Blackboard train wreck be on time today?” but I digress. During the following weeks we worked with Blackboard Support on various fixes. The use of Logstash, ElasticSearch, and Kibana really helped us find information and learn more about what was going on within our environment. We were able to find ActiveMQ and file locking issues quickly and discovered issues with session spray using our searches. (See some screen captures of Kibana below.)
The only negative thing I can say about Kibana is that it doesn’t have built in user management or the ability to alert users when specific data appears in the logs. GrayLog2 is an open source project that uses ElasticSearch and Logstash. However it does have the components that Kibana is missing. We might look at moving to it in the future. For right now, Kibana meets our needs.
In the end, these three tools have greatly improved how we can use data to address issues and fix changes within our Blackboard environment. I hope that it helps you when trying to decide how to empower your log file data.
Happy Back to School to everyone! I know that the start of the new school year creates many challenges and issues for Blackboard Learn Administration. For that reason I’m teaming up with my publisher, Packt Publishing, to give away a digital copy of my book Blackboard Learn Administration to three lucky winners!
If you haven’t read about the book, it covers many different topics. Including:
Create, install and update a Blackboard Learn instance
Create and manage courses with Blackboard Learn
Create and administer user accounts within Blackboard Learn
Brand Blackboard Learn with your organization’s other web presences
Integrate publisher building blocks and information systems within Blackboard Learn
Blackboard’s annual users conference kicked off with its technical and geeky side first as The 10th annual Blackboard Developers
From Left to Right: Mike McGarr, Stephen Feldman, David Ashman, and Gary Lang (Image from Blackboard)
Conference started with a day and a half of sessions focusing on four major tracks:
Extending Learn using Building Blocks
Cross-platform and the Cloud
It began with a morning keynote led by Stephen Feldman Vice President of Development, Performance and Security Engineering and Infrastructure at Blackboard and Mike McGarr Director of DevOps for Learn at Blackboard playing co-hosts at the conference’s keynote events. The opening keynote offered insight into the company’s push to implement a new directive from last year that leveraged building blocks to deliver and manage tools that have normally be built into the core.
The first big announcement was the move from the current edugarage.com website to developers.blackboard.com. Mark O’Neil has been heading up this project and the move over to the new URL won’t be swift, but will build on the important documentation and resources that many admins and developers have come to expect. The next announcement was the four different tracks within this year’s conference: Extending Learn using Building Blocks, Performance, Security, & Monitoring, Cross-platform & the Cloud, and Enterprise Integration. Gary Lang, the new Senior Vice President of Product Development took the stage. As of that day, he had been with the company only about ten days. Lang discussed his vision for Blackboard Learn and the creation of a “developer surface area” which will allow developers to integrate with all the different product platforms within the Company.
David Ashman, Chief Architect of Blackboard Learn, came to the stage next to talk about the work that the product team has been doing over the last year. If you have been under a rock or haven’t been following the changes within the Blackboard Learn product. The company has started to make changes to the way tools are integrated by using Learn’s APIs to make changes and improvements to the product without major upgrades. This gives institutions to implement the new tools without major downtime to upgrade all the code within the application. This is a big relief to many, however it still has its teething troubles as programmers have to increase what APIs are available and how building blocks can communicate with the product and other building block tools. In the end, the opening keynote really launched the conversations a bit more for attendees.
Jeremy Wiebold talks with attendees during a presentation. (Image from Blackboard)
One of the best presentations, in my humble opinion, was Nick McClure’s presentation to beginning Blackboard admins (see the presentation online thanks to Blackboard & Echo360). I have always said that empowering and educating the system administrators and support teams for Blackboard Learn instances greatly improves the end users experiences with local support and helps Blackboard’s support personnel quickly diagnose the issues that they face. This is one presentation that should be a requirement for every new admin and should be repeated every year.
The next session was my co-presentation with Jeremy Wiebold about the SIS Integration at the University of Missouri (Presentation Video). The room was much bigger than what I expected and our session was well attended. We got a lot of questions about how we made the move from our snapshot environment to the new SIS Frameworks. There were a few other presentations about SIS Frameworks at DevCon and Blackboard World, each presenter having their own spin on the topic which gave attendees a wide range of views and opinions on how to implement SIS Frameworks. Below is a copy of our slide deck from that presentation. We had a lot of questions from that presentation and I spent quite a bit of the afternoon discussing the SIS Framework process to other attendees.
After lunch, we then got the chance to hear Gene Kim, co-author of The Pheonix Project (Available on Amazon.com and the first 170 pages for free.)
and The Visible Ops Handbook (Available on Amazon.com) give our afternoon keynote. Gene has given many talks and he is an engaging and thought provoking speaker. Below you can find a sample of his presentation recorded by my VIP blogger and colleague Melissa Stange on her Google Glass. He also gave a similar speech during PuppetConf which can be found here on YouTube.
While I haven’t had time to start on the book. Reviews (available at Amazon.com) tell the too often repeated situation in IT departments of an urgent project, a set in stone date, and a lack of planning. The book’s narrative follows the failure and lessons learned from the project’s ashes. In the end, Kim’s point is that many IT Division find themselves in the middle of projects or implementations with timelines that don’t include all effected parties in the conversation. This creates products that have bugs, issues, and force teams to make shortcuts. In the end, Kim’s talk really touches to the need for more Kumbayah than silos within IT. Kim’s publisher gave away 100 copies of The Pheonix Project, one of those copies is mine and was signed by Kim.
The afternoon included a presentation by Matt Saltzman who is a member of Stephanie Tan’s Security team for the Blackboard Learn product. Both Stephanie and Matt are my security heroes when it comes to Blackboard Learn. My presentation was up against Stephanie’s this year so I missed hers (you can see it online), but I wasn’t going to miss Matt’s (See his presentation online). Matt discussed securing your Blackboard Learn environment. This was one of the chapter topics in my book. While Matt did go over a lot of what I discussed in the book, I was able to learn a bit more about port scanning and what information can be collected just from a port scan. He also helped me have more confidence in the recommendations that I made to readers. Which always makes an author feel a little better.
We ended the night with a few laps around the racetrack. The indoor track at Pole Position Raceway took me on a ride which I can say I’ve never experienced in my life. The event was enjoyable for most with video games, food, and an open bar (unless you were racing – no D&D). The evening ended early with a bus ride back to the hotel and prepping for the next day.
Day two started with much more collaboration. I found myself missing out on presentations to discuss options and issues with other attendees or catching up on some emails back at work. One presentation I did get to sit in on was the MBeans exposed! presentation (video available here) by Noriaki Tatsumi and Danny Thomas. They gave a great presentation on how to monitor the Blackboard application with MBeans and JConsole. These situations make me so thankful for the partnership with Blackboard and Echo360 who record many of the presentations given during DevCon and Blackboard World. It was great to see this tweet from Dan Rinzel (dan2bit) on Twitter.
The final keynote announced the winners of the Catalyst awards along with the introduction of a new track for next year, User Experience. It also included some rehashing of things discussed during the event. Mike and Steven both asked for more client presentations for next year’s event and announce the planning for an un-conference which should happen in Winter 2014. The event will be a virtual conference so more people can participate.
For some of you reading this post, this isn’t your first Blackboard World rodeo. For those new to the fold, users annually get together in what I call a cross between a conference, celebration, college homecoming, and alumni event which focuses on the many ways we use, maintain, and upgrade Blackboard products. I thought over the past few days about what my expectations are for the event, what attendees need to hear from Blackboard, and some don’t miss events during the week as well.
While Blackboard World (and the Client Appreciation Party) will clearly miss Blackboard’s former CEO and co-founder Michael Chasen, attendees welcome a new leader to the event as Jay Bhatt takes the reins. I know very little about Mr. Bhatt except from what is available in his biography. (and some googling) While we have come to learn the thoughts and ideals of CTO and President of Academic Platforms, Ray Henderson, I hope and expect that Mr. Bhatt will give attendees a clear understanding of what his vision is for the company and what it means to us.
Last year’s event in New Orleans contained many announcements, most were about the increasing acceptance of industry standards and tightening integration with publishers. I would believe that this continues to be a main focus during this year’s Blackboard World. The company’s wide variety of products and services require this type of integration. We should also get a preview of what new tools, technologies, and advancements the company is making in its different product lines.
Over the past few weeks my blog posts and listserv conversations voiced major concerns regarding the quality, improvement, and issues found in new versions in the Blackboard Learn product. Blackboard, with all due respect, can sometime have a hard time being innovative while ensuring a quality product. They aren’t alone in this struggle which includes most major software companies, however I have overheard admins look at releases and state “There’s more flash than substance.” (or just banging their heads into the table saying “They – missed – the -point -again.”) I’m not sure how, but the company needs to address these issues and concerns that seem to be growing to become the “gorilla in the room” before it turns from whispers into institutions walking away.
Blackboard Mobile Learn was introduced at Blackboard World 2009 and was welcomed by many as how Blackboard would bring its flagship product to the young mobile market. Smartphones and tablets have skyrocketed over the past few years, but Blackboard Learn’s Mobile Learn app has languished. Over the past four years, the application added many new options, but it doesn’t offer the ability to leverage all the product’s tools within Blackboard Learn. The company shows little interest in improving the web experience within the product if users don’t purchase the app (and the app doesn’t seem to be working to represent the complexity and flexibility of the product). I would love to see a mobile roadmap announcement with a projection to bring full functionality with every tool within Blackboard Learn either within the app or using a mobile browser. This is a challenge worthy of the company and the millions of mobile students and increasing numbers of mobile instructors.
2. Issues that must be addressed
While these expectations address what many users expect (or hope) to hear during the conference, there are some glaring issues that the company needs to face and address for its customers. First address the current product roadmap. Some in the Blackboard community have spoken out about issues and concerns with the current service pack / release program for Blackboard Learn. Some institutions can fall six or eight service packs behind the current release between upgrades. While Blackboard recommends upgrading to the latest and greatest version of their software, institutions state the lack of quality and stability in recent releases gives them reason to stay back and accept known issues and bugs within the application. Blackboard should use the conference to announce a major listening program made up of webex, surveys, along with face-to-face meetings to discuss how the company should support and address these concerns with measurable and attainable goals that will allow them to be held accountable.
The current patch, building block, and service pack programs can be a nightmare to understand. When an issue develops within the application, it is hard to know if the issue will be resolved in a patch, building block, or service pack, and the issues could be resolved in different ways for different versions. Blackboard must create a structure that will flag each bug, allow for investigation, then notify and post how each issue will be fixed for what versions. This could be something as simple as a matrix of versions and fix options. Developing a clear picture on how each issue will be addressed helps administrators and support staff at institutions quelch the pitchforks and torches. This process should then be publicized and feedback collected for continual improvement.
As the Blackboard Learn product line moves into the modularization of its tools, administrators need to see a consistent plan for these tool building blocks. If the company plans to continue to use them, they should fall into similar patterns and expected releases like service packs and cumulative patches. When an issue is a bug and the company has no plans to fix it, Blackboard Support personnel should give this information and state that if the institution feels this issue should be patched, they can create a comment in the case supporting their view with examples, if possible. This type of information provides support and product development the input to understand how tools within Blackboard Learn are used “in the field” while showing that the company is listening.
Ray Henderson, during his annual report at Blackboard World, measures and grades the successes of the company on many aspects, however the user community has little tangible evidence to the claims. Last year the company announced major purchases to improve product quality. During the year, administrators have heard from numerous company representatives stating that product quality continues to improve. It would be a welcome addition to Ray’s report card to have measurable analytical data that will back up these facts. A good example would be a table showing the number of known bugs (externally and internally reported) in all supported versions of Blackboard Learn. This information could be collected and kept within the Behind The Blackboard, but this can show how the company is improving the quality as newer versions come out. Even sharing estimates of how much programming work is being done to fix bugs in comparison to new product development would be great. No matter what happens, the vail that many see around the product’s quality needs to be lifted for those supporting the product.
3. Things I don’t want to miss…
My Blackboard World schedule of events has already started to become packed, but there are a few things that I wouldn’t want to miss. First are the three presentations I am participating in during DevCon and Blackboard World. Our SIS programmer, Jeremy Wiebold and I will be presenting twice about how we moved from using Blackboard Snapshot to the new SIS Frameworks at the University of Missouri. I am also co-presenting with Melissa Stange and Francesca Goneconti about Blackboard’s Social Tools feature. So please feel free to come to our presentations, but don’t miss out on many of the other great presentations during both events.
I also recommend that if you have questions, issues, or just need to talk about something that is going on with the Blackboard product you use, either reach out to someone within the Blackboard community or stop and ask one of the many employees at the Blackboard booths or during events and ask them if you can schedule some one-on-one time with them during the conference. I’ve found that Blackboard welcomes constructive feedback to improve the products and services they offer and you never know if your conversation will get the issue resolved. This type of interaction along with the discussions at feedback and user experience sessions really allow us as users give constructive feedback on how the product may work in future releases.
I’m leading a Birds of the Feather session on Thursday, that are great chances to talk with people about different topics many wonder about. These conversations help each user to network and gain insight into what did work and what might go wrong which greatly helps increase the knowledge of the user community as a whole. In general, I find myself spending a large amount of time during the conference discussing issues, scheduling one-on-ones with other attendees to really learn what other institutions are doing. Sometimes these conversations can form partnerships with institutions to create building blocks or share other data. In the end, it’s a win-win.
I hope to use this post to help me digest and review how Blackboard World went. During the conference, I will work to post a daily recap of each day, which (I hope) highlights the important aspects that you may have missed. Two or three weeks later, I hope to have my review posted here as well.
6/12 Update: John Porter announced via a Blackboard Admin ListServ that global patches which require all Blackboard services stopped, will be removed from cumulative patches. The ETA for the implementation of this plan hasn’t been revealed.
At Blackboard World 2012, Ray Henderson stood before a packed audience and proclaimed that Blackboard Learn would start to create tools as building blocks. This announcement was welcomed by many administrators. It promised the ability to add new tools and improve them with the installation of a building block instead of downtime associated with an upgrade.
The release of Service Pack 10 in late 2012 fulfilled this promise by making the content editor, SIS Frameworks, authentication integrations, and several course tools into building block modules. The new Software Updates module in the System Admin tab allowed the company to push new building blocks and building block updates to admins and install them with some efficiency.
Sadly when the rubber meets the road, there are some concerns. It seems that administrators (like myself) have found that when we discover bugs within these tools or integrations, we are given long planned fixes, which are several service packs out. In the past two weeks on Service Pack 11, I personally have found issues within the discussion board, content editor, and SIS Frameworks. While I will admit that some of the issues might not be completely embedded into a tool’s building block, it seems that the expectations from system administrators for quick bug fixing and resolution to building block issues have fallen below expectations only a year after they were promised.
Another improvement, while not announced at last year’s Blackboard World, was the implementation of the Blackboard Patch Utility and Blackboard Patch Repository. This tool and service work together with the Software Updates module to notify and update current Blackboard Learn instances with the latest patches to known issues. Sometimes important or critical patches are combined by Blackboard Support into Cumulative Patches. Over the past year, these patches have started to come out every other week on a regular basis. This is great for system administrators, it gives us a plan on how and when we will get critical fixes to the systems we support.
However the Cumulative Patching process does have its drawbacks. The concept of cumulative patches is that each patch builds on the other. For example, Cumulative Patch 4 includes the patches that were also in the past three. This means that if a specific patch within Cumulative Patch 1 needs to have every server down to address an issue, each forthcoming one will require the entire system down, which can make a system admin’s life just a little bit more difficult. Another issue is the increasing size of the Cumulative Patch itself which to my surprise included multiple building block installations which I would have thought could be delivered with the aforementioned Software Updates. The installation of these building blocks during the initial patching of my Blackboard Learn instance took three times as long as patching the other application nodes. My hope is to see more improvements in the delivery and development of the cumulative patch process within Blackboard.
In the coming Blackboard World 2013 keynotes and discussions of where the company and its products have been, I hope that Ray’s review will address this. We, the system administrators of Blackboard Learn systems around the world, can feel a little jaded by the lack of progress on these topics. Let’s hope that Blackboard can refocus on fulfilling the promises made in New Orleans before rolling the dice on more of them in Las Vegas.
I'm a Blackboard Learn Administrator that designs and creates excellent online education environments that meet student, faculty, and institution needs. I'm a Blackboard Catalyst Award Winner and the author of Blackboard Learn Administration from Packt Publishing. I enjoy photography, video games, & cooking.