I have spent the past 10+ years working with Blackboard as a system administrator at multiple higher education institutions in the US. In 2009, Blackboard presented me the Blackboard Catalyst Award for my work within the Blackboard Community. In 2013, I authored the first technical book for Blackboard Learn Administrators published by Packt Publishing.
Here is the place I post my musings, issues, and finding as I work through the sometimes frustrating, humorous, but always interesting world of Blackboard.
Note: I apologize for the delay in getting this post up. It has been a crazy time, but more on that in a later blog post.
Zero. None. Null. Zip. Nada. Zilch. Wouldn’t it be nice to have that many user complaints or questions about your Blackboard Learn instance? I think that an empty inbox would make me wonder if the devil himself had made it to Macy’s for a winter coat sale. It can be hard to play defense with a Blackboard Learn environment. I think most Blackboard Admins would like to be slightly ahead of that snowball (or worse) that’s rolling down the mountain behind us. I want to share with you some of the options available to allow you to attack issues before they hit your inbox.
Companies offer a variety of software packages to monitor issues within applications like Blackboard Learn. In this discussion, I’m going to focus on the end user and how he or she receives and relates to the data within Blackboard. Many users might think that this type of software will be a hard sell to administration and I can understand why. Many times, the end user seems to be the hardest part to troubleshoot due to the external variables and if your organization doesn’t monitor this area it can be hard to determine there is a problem.
My organization, The University of Missouri, didn’t have an understanding of our users. We really lacked data about our Blackboard Learn environment. One of the first items I wanted to address when I came to campus was to find a way to collect data to better understand the application and it’s users. We started by looking at Google Analytics. The free version gave us a lot of what we might want to use. However there were concerns about the data being stored on Google’s servers and that based on the amount of data within the Blackboard database; we would quickly have to pay to use Google Analytics.
I started to lose hope that we might find the right product to collect data about Blackboard. Then I was directed to Piwik (piwik.org). I simply describe it as an open source version of Google Analytics on steroids. It addressed our issues with data since the application stores information in a local MySQL database and can handle the amount of traffic we regularly get.
Piwik allows us to see when loads are highest and lowest. This has been a big help in planning maintenance windows. We thought the highest usage of Blackboard was during midterms, however we found out this was wrong. The highest usage, according to Piwik, was the fifth week of the semester. Specifically, monday through wednesday of that week. Below you can see usage over two weeks.
Information like this can be helpful, but many times we need to see more. An example would be what type of operating systems and browsers are used to access Blackboard. Below are two examples; the usage of Apple products greatly surprised us and our administration. With this information we began to test more of our tools on Macs and with Mac-based browsers.
We also can find the user’s location. This information has been great to show the impact of our environment not just within Missouri, the Midwest, and the United States, but also worldwide. On this world map below, the darker the blue, the more users from that country.
While the information we have learned is good, we can gain more by using Piwik. My favorite option displays the views for a variety of pages. Each is based on the URL, so we get a broad picture. We can see usage, average time on page, and my favorite, average generation time. The last one helps understand if there are performance issues with a specific tool when comparing it with others. See the table below – it’s an example of what Piwik can bring to light.
As you can see, Blackboard environment has a lot of information locked away. Piwik offers administrators the key to gain access to better understand not only your system, but your users as well. Knowing your users’needs can put you on the offense to address issues and problems out of the inbox and you relaxing with no snowball in sight.
One of the many topics that my book, Blackboard Learn Administration, talks about is upgrading and installing Blackboard Learn. However I didn’t really go into detail about the process in preparing to upgrade or install a new version of Blackboard Learn for your institution. Well this blog post will hopefully erase that neglect. Let’s review how I prepare for a Blackboard Learn upgrade.
Select a version
Blackboard Learn has been rather stable in their release structure. Recently they announced the move from four releases a year to two releases a year. This might be a good and bad option for institutions depending on the release date and maintenance schedule. I try to recommend the latest version released prior to our testing window. The testing and security scans can take weeks. This along with implementing the new version means it will be about four to five months old by the time it’s put on our production environment.
The biggest concern deals with the amount of time Blackboard will support the version with regular patches for security and other issues that are found. Recently, Blackboard has moved to supporting versions for 18 months after their release. I like this plan because it gives the institution a guaranteed year of support with a version. I’m fond of the yearly upgrade. Some might disagree with this plan, but it seems to be the best based on the following factors.
A yearly upgrade gives a lot of time for security and user testing.
While the selected version is supported and patched, it doesn’t utilize “bleeding edge technology”
Reduces training and feature fatigue for faculty, staff and students.
Additional resources from Blackboard and the community are available
An upgrade can require a lot more than just installing new applications. After many years of performing Blackboard Learn upgrades, I’ve learned that all the parties possibly impacted by an upgrade should take part in upgrade planning, whether that would be the support staff, database administrators, networking teams, server administrators, or security teams. These people may be called on by you, the Blackboard administrator, to help with planning the upgrade or making needed changes. I’ve found a Gantt chart useful to show the upgrade plan. I hand this out at a meeting with all these parties. It’s a big help to get everyone on the same page and have a schedule that we all know and can use when planning this or other projects. I’ve also taken to scheduling monthly meetings with all these stakeholders. While I do cancel some of them, it still gives a regular schedule to discuss changes or issues that are coming up that could influence the planned upgrade.
During the first meeting I just mentioned, one of the topics we talk about is upgrading not only Blackboard, but the supporting software. An upgrade is a great time for server and database administrators to apply security patches or move to a new operating system or application software supported by Blackboard. With our version already chosen, we can look at what software supports the application and quickly plan for this. We might also plan to implement additional hardware components such as load balancers or new IP configurations. These types of changes can be added to our Gantt chart and the update pushed to all affected parties.
When preparing for an upgrade, not does it include changes to the latest version of Blackboard. It will probably include new tools or 3rd party integrations, even editing tool names or improving the application’s customized layout for some. These other items could create more havoc for users if they add to an upgrade without planning.
This is why our team creates a set Blackboard configuration. This means that we all agree early in the upgrade testing process what tools should be installed, what items will be upgraded, enabled, etc. This helps the testing, security, and documentation teams know what the official changes will be. While the set configuration may change, this gives the teams a good place to start and allows for possible changes to be brought to the meetings I mentioned earlier.
The biggest challenge when preparing for an upgrade, in my opinion, is user testing. Does everything still function as we expect in the new version of Blackboard Learn? What the best way to tackle this? The Educational Technologies at the University of Missouri has a team of students who work with Blackboard and we have them help us in our user testing. Each year I create paper copies of several tasks that we expect to do in Blackboard. Each task list becomes associated to an operating system and browser supported by Blackboard. A few of these are available below to review. The student picks up the task list and, using the browser and operating system listed on the front, can complete the tasks listed. They sign off on each task either stating that it works or not. If it doesn’t work, the student writes out an explanation that will help our team figure out what’s going wrong.
In the end, an upgrade plan can help organize the move to a new Blackboard version. It also provides planning for additional services that can impact your upgrade. I recommend picking a version that will give you a few months of testing and do the following:
Look at what upgrade version will meet your needs based on topics such as how long is it supported by Blackboard, etc.
Bring all affected parties to the table, tell them, keep them on schedule.
Find what hardware and software changes can be made to improve support of the Blackboard application
Have all interested parties agree to a set Blackboard configuration for the upgrade for testing purposes.
Find available options to have users test tasks within different browser and operating system combinations supported with the new version.
Best wishes to your (hopefully) successful upgrade!
Greetings everyone! Sorry that I haven’t been able to post but I have recently been dealing with a Blackboard production environment that has spent much of its time in various rings of hell. This has happened on and off since the start of the semester (about 14 weeks ago as of the writing of this post). This has been a very difficult time, but there has been one ray of sunshine for us, our implementation of Kibana, ElasticSearch, and Logstash proved invaluable during troubleshooting.
In my opinion, these three applications are the holy trinity of logging for any application like Blackboard Learn. They allow an admin to unlock the information kept in logs at a cost that most IT budgets can handle (the three tools are open source). I learned about these tools from our central server group (CSG) who had heard about the tools during PuppetCon. (See the video below)
They had started testing them in their “skunkworks” area. During a meeting, I lamented the difficulty searching log files on six different application servers. These difficulties made me create the log collection script which I blogged about earlier this year. One of the team members suggested Logstash, ElasticSearch, and Kibana.
The next week I was at my server administrator training at Blackboard HQ in Washington, DC. During that week, one of the attendees from SUNY mentioned these tools as well. These two discussions had sold me. Back on campus, I quickly moved to test out how helpful and easy it would be to implement these applications within our Blackboard Learn environments.
The three tools Kibana, ElasticSearch, and Logstash all have different parts that they play in the log puzzle.
Logstash – This application brings the logs and event data from one or many systems to one location. The open source product runs on top of ElasticSearch.
ElasticSearch – This application is the engine of Logstash. It collects, indexes, parses, and searches the log data.
Kibana – The frontend of Logstash and ElasticSearch, this application is the Google search for all your logs. The user can be as basic or as advanced in how they want to search to find log information. The latest release of Kibana adds more visuals based on collected data.
We were already using shared network storage to hold all the log files from our production, quality assurance, and development instances. This meant that 95 percent of our files wouldn’t be difficult to collect. We could simply mount the shared storage to collect the data. We then had to use a shipper for the other 5 percent that don’t sit with the network storage, such as tomcat logs. The shipper runs on the application server and sends all data written to that file for indexing and storage. We spent a few weeks making sure that the shipper didn’t put any additional load on the application servers. Next we had to learn how to grok.
Grok is a pattern set used by the Logstash application to breakdown logging input and make it searchable. (See Grok Patterns) We created multiple fields for items such as email subject lines, IP addresses, etc. that appeared in the logs. The fields gave us the ability to leverage the data collected and the application’s indexing abilities.
Once all the data was being collected, groked (or parsed), and indexed. The Kibana application stepped into the spotlight. While Kibana has a simplistic interface, it is powerful and sometimes cryptic. I had to create a Kibana Cheat Sheet for myself and my team on how to use the web application to find logging data.
Once implemented, a few team members started to use the information to collect data about our environment. However our fall semester started out with our environment crashing on a regular basis, nearly as well scheduled as a subway train schedule. The only humor was that we would regularly be able to call the time for these failures. “Will the 11:03 Blackboard train wreck be on time today?” but I digress. During the following weeks we worked with Blackboard Support on various fixes. The use of Logstash, ElasticSearch, and Kibana really helped us find information and learn more about what was going on within our environment. We were able to find ActiveMQ and file locking issues quickly and discovered issues with session spray using our searches. (See some screen captures of Kibana below.)
The only negative thing I can say about Kibana is that it doesn’t have built in user management or the ability to alert users when specific data appears in the logs. GrayLog2 is an open source project that uses ElasticSearch and Logstash. However it does have the components that Kibana is missing. We might look at moving to it in the future. For right now, Kibana meets our needs.
In the end, these three tools have greatly improved how we can use data to address issues and fix changes within our Blackboard environment. I hope that it helps you when trying to decide how to empower your log file data.
Happy Back to School to everyone! I know that the start of the new school year creates many challenges and issues for Blackboard Learn Administration. For that reason I’m teaming up with my publisher, Packt Publishing, to give away a digital copy of my book Blackboard Learn Administration to three lucky winners!
If you haven’t read about the book, it covers many different topics. Including:
Create, install and update a Blackboard Learn instance
Create and manage courses with Blackboard Learn
Create and administer user accounts within Blackboard Learn
Brand Blackboard Learn with your organization’s other web presences
Integrate publisher building blocks and information systems within Blackboard Learn
Blackboard’s annual users conference kicked off with its technical and geeky side first as The 10th annual Blackboard Developers
From Left to Right: Mike McGarr, Stephen Feldman, David Ashman, and Gary Lang (Image from Blackboard)
Conference started with a day and a half of sessions focusing on four major tracks:
Extending Learn using Building Blocks
Cross-platform and the Cloud
It began with a morning keynote led by Stephen Feldman Vice President of Development, Performance and Security Engineering and Infrastructure at Blackboard and Mike McGarr Director of DevOps for Learn at Blackboard playing co-hosts at the conference’s keynote events. The opening keynote offered insight into the company’s push to implement a new directive from last year that leveraged building blocks to deliver and manage tools that have normally be built into the core.
The first big announcement was the move from the current edugarage.com website to developers.blackboard.com. Mark O’Neil has been heading up this project and the move over to the new URL won’t be swift, but will build on the important documentation and resources that many admins and developers have come to expect. The next announcement was the four different tracks within this year’s conference: Extending Learn using Building Blocks, Performance, Security, & Monitoring, Cross-platform & the Cloud, and Enterprise Integration. Gary Lang, the new Senior Vice President of Product Development took the stage. As of that day, he had been with the company only about ten days. Lang discussed his vision for Blackboard Learn and the creation of a “developer surface area” which will allow developers to integrate with all the different product platforms within the Company.
David Ashman, Chief Architect of Blackboard Learn, came to the stage next to talk about the work that the product team has been doing over the last year. If you have been under a rock or haven’t been following the changes within the Blackboard Learn product. The company has started to make changes to the way tools are integrated by using Learn’s APIs to make changes and improvements to the product without major upgrades. This gives institutions to implement the new tools without major downtime to upgrade all the code within the application. This is a big relief to many, however it still has its teething troubles as programmers have to increase what APIs are available and how building blocks can communicate with the product and other building block tools. In the end, the opening keynote really launched the conversations a bit more for attendees.
Jeremy Wiebold talks with attendees during a presentation. (Image from Blackboard)
One of the best presentations, in my humble opinion, was Nick McClure’s presentation to beginning Blackboard admins (see the presentation online thanks to Blackboard & Echo360). I have always said that empowering and educating the system administrators and support teams for Blackboard Learn instances greatly improves the end users experiences with local support and helps Blackboard’s support personnel quickly diagnose the issues that they face. This is one presentation that should be a requirement for every new admin and should be repeated every year.
The next session was my co-presentation with Jeremy Wiebold about the SIS Integration at the University of Missouri (Presentation Video). The room was much bigger than what I expected and our session was well attended. We got a lot of questions about how we made the move from our snapshot environment to the new SIS Frameworks. There were a few other presentations about SIS Frameworks at DevCon and Blackboard World, each presenter having their own spin on the topic which gave attendees a wide range of views and opinions on how to implement SIS Frameworks. Below is a copy of our slide deck from that presentation. We had a lot of questions from that presentation and I spent quite a bit of the afternoon discussing the SIS Framework process to other attendees.
After lunch, we then got the chance to hear Gene Kim, co-author of The Pheonix Project (Available on Amazon.com and the first 170 pages for free.)
and The Visible Ops Handbook (Available on Amazon.com) give our afternoon keynote. Gene has given many talks and he is an engaging and thought provoking speaker. Below you can find a sample of his presentation recorded by my VIP blogger and colleague Melissa Stange on her Google Glass. He also gave a similar speech during PuppetConf which can be found here on YouTube.
While I haven’t had time to start on the book. Reviews (available at Amazon.com) tell the too often repeated situation in IT departments of an urgent project, a set in stone date, and a lack of planning. The book’s narrative follows the failure and lessons learned from the project’s ashes. In the end, Kim’s point is that many IT Division find themselves in the middle of projects or implementations with timelines that don’t include all effected parties in the conversation. This creates products that have bugs, issues, and force teams to make shortcuts. In the end, Kim’s talk really touches to the need for more Kumbayah than silos within IT. Kim’s publisher gave away 100 copies of The Pheonix Project, one of those copies is mine and was signed by Kim.
The afternoon included a presentation by Matt Saltzman who is a member of Stephanie Tan’s Security team for the Blackboard Learn product. Both Stephanie and Matt are my security heroes when it comes to Blackboard Learn. My presentation was up against Stephanie’s this year so I missed hers (you can see it online), but I wasn’t going to miss Matt’s (See his presentation online). Matt discussed securing your Blackboard Learn environment. This was one of the chapter topics in my book. While Matt did go over a lot of what I discussed in the book, I was able to learn a bit more about port scanning and what information can be collected just from a port scan. He also helped me have more confidence in the recommendations that I made to readers. Which always makes an author feel a little better.
We ended the night with a few laps around the racetrack. The indoor track at Pole Position Raceway took me on a ride which I can say I’ve never experienced in my life. The event was enjoyable for most with video games, food, and an open bar (unless you were racing – no D&D). The evening ended early with a bus ride back to the hotel and prepping for the next day.
Day two started with much more collaboration. I found myself missing out on presentations to discuss options and issues with other attendees or catching up on some emails back at work. One presentation I did get to sit in on was the MBeans exposed! presentation (video available here) by Noriaki Tatsumi and Danny Thomas. They gave a great presentation on how to monitor the Blackboard application with MBeans and JConsole. These situations make me so thankful for the partnership with Blackboard and Echo360 who record many of the presentations given during DevCon and Blackboard World. It was great to see this tweet from Dan Rinzel (dan2bit) on Twitter.
The final keynote announced the winners of the Catalyst awards along with the introduction of a new track for next year, User Experience. It also included some rehashing of things discussed during the event. Mike and Steven both asked for more client presentations for next year’s event and announce the planning for an un-conference which should happen in Winter 2014. The event will be a virtual conference so more people can participate.
I'm a Blackboard Learn Administrator that designs and creates excellent online education environments that meet student, faculty, and institution needs. I'm a Blackboard Catalyst Award Winner and the author of Blackboard Learn Administration from Packt Publishing. I enjoy photography, video games, & cooking.
Order My Book…
Blackboard Learn Administration By: Terry Patterson