6/12 Update:  John Porter announced via a Blackboard Admin ListServ that global patches which require all Blackboard services stopped, will be removed from cumulative patches. The ETA for the implementation of this plan hasn’t been revealed.

At Blackboard World 2012, Ray Henderson stood before a packed audience and proclaimed that Blackboard Learn would start to create tools as building blocks. This announcement was welcomed by many administrators. It promised the ability to add new tools and improve them with the installation of a building block instead of downtime associated with an upgrade.

The release of Service Pack 10 in late 2012 fulfilled this promise by making the content editor, SIS Frameworks, authentication integrations, and several course tools into building block modules. The new Software Updates module in the System Admin tab allowed the company to push new building blocks and building block updates to admins and install them with some efficiency.

Sadly when the rubber meets the road, there are some concerns. It seems that administrators (like myself) have found that when we discover bugs within these tools or integrations, we are given long planned fixes, which are several service packs out. In the past two weeks on Service Pack 11, I personally have found issues within the discussion board, content editor, and SIS Frameworks.  While I will admit that some of the issues might not be completely embedded into a tool’s building block, it seems that the expectations from system administrators for quick bug fixing and resolution to building block issues have fallen below expectations only a year after they were promised.

Another improvement, while not announced at last year’s Blackboard World, was the implementation of the Blackboard Patch Utility and Blackboard Patch Repository. This tool and service work together with the Software Updates module to notify and update current Blackboard Learn instances with the latest patches to known issues. Sometimes important or critical patches are combined by Blackboard Support into Cumulative Patches. Over the past year, these patches have started to come out every other week on a regular basis.  This is great for system administrators, it gives us a plan on how and when we will get critical fixes to the systems we support.

However the Cumulative Patching process does have its drawbacks. The concept of cumulative patches is that each patch builds on the other. For example, Cumulative Patch 4 includes the patches that were also in the past three. This means that if a specific patch within Cumulative Patch 1 needs to have every server down to address an issue, each forthcoming one will require the entire system down, which can make a system admin’s life just a little bit more difficult. Another issue is the increasing size of the Cumulative Patch itself which to my surprise included multiple building block installations which I would have thought could be delivered with the aforementioned Software Updates. The installation of these building blocks during the initial patching of my Blackboard Learn instance took three times as long as patching the other application nodes. My hope is to see more improvements in the delivery and development of the cumulative patch process within Blackboard.

In the coming Blackboard World 2013 keynotes and discussions of where the company and its products have been, I hope that Ray’s review will address this.  We, the system administrators of Blackboard Learn systems around the world, can feel a little jaded by the lack of progress on these topics. Let’s hope that Blackboard can refocus on fulfilling the promises made in New Orleans before rolling the dice on more of them in Las Vegas.

We’ve spent months and months at the university building up our users to the new features in Service Pack 11.  Especially the new content editor, however over the past few weeks we’ve found issues within SP11 that cause myself and some of the support staff to ask all local hardware stores to stop selling tar and pitchforks. The worst part is that some of the responses from Blackboard concern me.

Issue #1 – The inability to change points possible for assessments within Grade Center.

Synopsis - In Service Pack 8, instructors could go into an assessment’s grade column within the grade center and change the points possible.  In Service Pack 10 and later, this option has been removed (by graying out the points possible option).

Issue - Multiple instructors on our campus use random blocks to pull test questions for assessments. Here’s an example. An instructor pulls 32 questions using a random block with each question being one point. However the instructor wants the assessment to only count for 30 points instead of 32 points. The instructor uses this option because they want to allow the students to have two “mulligans” in case they have issues with displaying or understanding two of the questions within the assessment. This issue has affected multiple courses and programs within our campus and has also been recognized as a major issue for other colleges within our system.

Blackboard’s Response - After creating a ticket, the issue was defined that the problem was “functioning as designed” which is very frustrating for our users. We plan to ask this as an enhancement request and request it be addressed immediately.

Issue #2 – Extra line breaks appear within emails using the Blackboard Learn email tool and new VTBE.

Synopsis - When an instructor uses the Blackboard Email tool, extra line breaks <br> appear within the email when the user gets the email.  These don’t appear in the source code that the user put into the course.  Here’s an example of the original HTML code:

<p>Hello Botanists!</p>

<p>Bless the students who start on things early.&nbsp; Mitchell and I had planned to meet on Collaborate earlier this week, but had technical problems.&nbsp; So this is a warning that there have been some recent changes in Java this spring and Collaborate (the software

you’ll use to connect with me this morning from 9-10 for office hours) depends on Java.&nbsp; So, I recommend the following things:</p>

<ol>

<li>Give yourself PLENTY of time (20&#43; minutes) to try getting on Collaborate to make sure it is working for you.</li>

<li>If it is not working for you, then update your Java for free at: <a href=”http://java.com/en/download/index.jsp”>

http://java.com/en/download/index.jsp</a> and try again.</li>

<li>Try another web browser.&nbsp; I have better luck with Firefox than with Safari.<br>

</li></ol>

<p>This software is worth the trouble!&nbsp; You’ll probably use it in other classes in the near future.&nbsp; This semester it is a great way to meet online with your group members when you want to talk about your semester-long project.&nbsp; The &quot;Coffee Shop&quot; Collaborate session is always open for your use!&nbsp; Of course, it is also how/where I hold my office hours in online classes.&nbsp; To meet me for office hours, click on the &quot;Office Hours&quot; session.</p>

<p>Best wishes,</p>

<p>Bethany</p>

The email HTML looks like this:

<meta http-equiv=”Content-Type” content=”text/html; charset=utf-8″>

<p>Hello Botanists!</p>

<br>

<p>Bless the students who start on things early.&nbsp; Mitchell and I had planned to meet on Collaborate earlier this week, but had technical problems.&nbsp; So this is a warning that there have been some recent changes in Java this spring and Collaborate (the software you’ll use to connect with me this morning from 9-10 for office hours) depends on Java.&nbsp; So, I recommend the following things:</p>

<br>

<ol>

<br>

<li>Give yourself PLENTY of time (20&#43; minutes) to try getting on Collaborate to make sure it is working for you.</li><br><li>If it is not working for you, then update your Java for free at: <a href=”http://java.com/en/download/index.jsp”> <a href=”http://java.com/en/download/index.jsp”>http://java.com/en/download/index.jsp</a></a> and try again.</li>

<br>

<li>Try another web browser.&nbsp; I have better luck with Firefox than with Safari.</li>

<br>

</ol>

<br>

<p>This software is worth the trouble!&nbsp; You’ll probably use it in other classes in the near future.&nbsp; This semester it is a great way to meet online with your group members when you want to talk about your semester-long project.&nbsp; The &quot;Coffee Shop&quot; Collaborate session is always open for your use!&nbsp; Of course, it is also how/where I hold my office hours in online classes.&nbsp; To meet me for office hours, click on the &quot;Office Hours&quot; session.</p>

<br>

<p>Best wishes,</p>

<br>

<p>Bethany</p>

Issue - Adding extra white space can create issues for users who have low screen resolutions on their computers. The white space can indicate the end of an email to a reader, however the user has much more information. Further it is disturbing to the sender that the Blackboard application is injecting additional HTML code without displaying the change. Also while the Shift-Enter option might be a workaround, we were told that this content editor would fix many of the issue with the old VTBE editor. Instead it is causing the same issues or creating new ones.

Blackboard’s Response - “The patch request for the issue with the email spacing has been denied because the extra spacing does not interfere with critical functionality. You can use the Shift Enter keys for line breaks to avoid the extra spacing. This is scheduled to be corrected in 9.1 service pack 15.”

There still remain two issues that I still can’t replicate fully or haven’t opened a support ticket.  The first is the disappearance of replies to discussion board posts with special characters within them. The second is the SIS Framework ignoring data source keys for enrollments. We have a student who was accidentally enrolled in a course and then was disabled when he dropped it. He still was taking the course to finish an incomplete from the previous semester. We changed his disabled enrollment from one data source key to the system data source key and when the SIS store was ran, it changed the data source key and row status back. Snapshot would ignore any enrollments that weren’t assigned to its data source key.

These are just a few of the issues we have found so far.  I welcome any comments or suggestions about them.

“Pardon the rant, but every time I open a ticket with Blackboard Support it seems they always need another piece of information to investigate the issue discussed in the ticket. I spend days in a back and forth conversation with a support person asking for different files. This wastes precious time, and delays a response to finding a solution. In the end, I get emails from the effected user asking for an update. I become frustrated as well as the user. I need to find a new way to resolve end this lag time.”

That was the rant I had with a colleague a few months back. It seemed that it was time to find a new way to resolve this issue. So I thought that if I could write a script that collects commonly requested information by Blackboard Support, then have it zipped into a file which I could then attach to the case and save some time. I knew that this script would need to collect a lot of information, so I would need to learn what information it needed to collect. I talked with Blackboard Support and they suggested that I collect several different files.  I organized the files requested by support into five categories that should address the different issues we commonly face.

Configuration

This file set includes the following files:

• bb-config.properties
• bb-config.properties.template
• authentication.properties

These files should answer most of the questions about how the Blackboard Learn instance is configured.

Performance

This file set contains the following files:

• bb-access-log from the logs/tomcat folder
• stdout-stderr-log from the logs/tomcat folder
• catalina-log.txt from the logs/tomcat folder
• bb-services-log.txt
• along with the configuration files we just mentioned

To diagnose a performance issue it is also import and to collect  the output from several diagnostic tools, these are collected and included in the file set by the script:

• a perf_report
• output from the vmstat command
• four thread dumps
• top commands ran every thirty seconds.

Patch and Updates

This file set contains the following files:

• All the files from the update-tools log folder
• bb-installer-log.txt file
• A list of all patches that have been applied to the server

Course Copies, Archives, and Restores

This file set contains:

• The content-exchange-log.txt from the content-exchange folder

SafeAssign

This file set has these files:

• bb-services-log.txt
• sa.properties
• all safeassign logs

These files and output data were described by Blackboard Support as very important to getting issues addressed and quickly start work on most Blackboard Support tickets. Once I had this information, I then created a script that would collect this data for me, or anyone that needed to open a ticket with Blackboard Support about our instance. In my script, there are several things that needed to happen.

1. When the script runs on a node, it requests an email address. This email address must match a specific domain before it will be allowed to move on. If it doesn’t match, the script errors out, this prevents sensitive data from being sent to an incorrect address.
2. Once the email address is accepted, the script creates a folder within the logs directory of our blackboard application server. This is where the files will be copied to create a zip file later.
3. Within this folder each category gets its own subdirectory where the files will go.
4. The script then starts to copy the files for the configuration and performance directories.
5. The script now runs the performance report, top command, vmstat command along with the four thread dumps and zips them into a file before moving the zipped file to the performance folder and continuing to copy performance files.
6. Then the script copies the information for the patches and updates directory
7. Then it copies the files for course copies, restores, and archives along with safassign files into their directories.
8. It adds a readme.txt file that allows the support person to know what information is within the subdirectories.
9. The system will zip the directory up and email it as an attachment to the email address given earlier.
10. I can download the attached file and then add it to the case.

This script has saved me a lot of time and energy, especially when I have to collect these logs on all of our production nodes. When Blackboard Support requests this information, I always tell them it’s attached in the first comment. Which normally surprises,  and I hope pleases, them. Please feel free to use or adapt this in your Blackboard Learn environment. Below are the readme.txt and bb_log_collection.txt script files for Linux. Sorry for Windows users, but you can look at the information here and hopefully create a script for the Windows OS, and even if a person doesn’t want to create a script, the common files should at least give you a good head start.

Download the ReadMe.txt file
Download the bb_log_collection.txt file

Technically Yours,

Blackboard Guru

This review is by Guy Wilson, Educational Technology Specialist at the University of Missouri.  Many thanks for his work on reviewing this application.   – Terry

Like many apps, the newest version of Blackboard Mobile Learn (BBML) is a mix. In this case with one well done new feature (integration), bug fixes, and some disappointing omissions. The most noticeable (and infamous) difference, though, is not a software feature, but the change in its financing model.

 
Previous to version 3.1, Blackboard had partially financed the app through an agreement with Sprint. While this kept the app free to end users, it placed limitations on Blackboard Mobile Learn’s connectivity options. Earlier Android versions could only connect through the 3G data plan if they were Sprint subscribers, while iOS devices could only connect through WiFi. (The agreement predated the Sprint-Apple iPhone deal.) This had always been an Achilles heel, limiting Android use to a fraction of iOS use on our campus.

The new BBML 3.1 no longer has ties to any carrier. Users can access it over WiFi or any cell network, so long as they are on Android or iOS. The trade-off is that they now have to make an in-app purchase before using the app ($1.99 for one year or $5.99 for “unlimited” access). Complaints have been numerous, some of them bitter, and not without foundation. Blackboard might have gained a better reputation by pursuing a freemium model, allowing users to unlock advanced features with in-app payments, while using the basic functionality.

The best feature of 3.1, and one that probably makes the price worthwhile for faculty on the go, is Dropbox integration. Blackboard has executed this almost flawlessly. Assuming that Dropbox is already set up on the device, users only have to click the settings icon, choose Link Dropbox, and grant the app permission to access the user’s Dropbox account. Once setup, using Dropbox is straightforward, in those tools that can make attachments, one need only click the Attachments button. Three icons, one for the camera, one for Photos, and one for Dropbox appear on iOS. On Android, the choices are Dropbox, Local File System, and Other (which allows opening files from various apps and online sources). Click Dropbox, and a file browser opens, choose one or more files, and then click Add.

Instructors can use this feature to create items and add attachments, which has been a major problem for iPad and iPhone faculty for some time. Students and faculty can also use this feature to add attachments to discussion, blog, and journal postings. (Why this was not extended to wikis, assignments, and SafeAssignments is a perplexing mystery.) Anywhere  attachments appear, users can save them to their Dropbox accounts.

As good as this feature is, comparison between the iOS version and the Android version reveals a major conceptual problem in the app. Basically, this is a feature that would have made sense in iOS in 2010, but not in late 2012. Most iOS apps that can open and close files can now exchange files with other apps. Consider this scenario. A student creates a long attachment in a productivity app that needs to be uploaded to a blog post. To get it into Blackboard without resorting to a desktop or laptop, she has to first send it Dropbox, then use the attachment procedure described above in BBML. If BBML could exchange files with other apps, she could use an iOS “Open In” command to send it to BBML, which could store it in a cache until it is wanted. This approach has been implemented in other apps (e.g., iThoughts HD) and would be a welcome improvement. Until something along these lines is implemented in the iOS version, Android users have a real advantage.

Another surprising omission in some ways, reflecting the app’s student orientation and missing advanced functionality for instructors, is the lack of access to Blackboard’s file system and content collections. Instructors who need access to the files have to use a browser. If they want to upload or download content on an iOS device, they are limited to using WebDAV, which requires a separate app, like GoodReader or one of Apple’s iWorks apps. (This is also the only way to upload a document larger than 10MB to Blackboard on iOS.)

While Dropbox integration is a welcome addition to the BBML app, it does little to improve the functionality needed by students and instructors alike.  The past two years since the app’s introduction have seen some stylized changes and the addition of a mobile test creation tool, leaving many of the major tools with Blackboard Learn unintegrated.  A great example for students would be the lack of an option to submit an assignment.  While integration between tools like Dropbox might put developers closer to making this option a reality.  Students already utilize iPads and Android devices to create and edit assignments.  Instructors lack the ability to access the Grade Center using the BBML, an issue that our institution would like to see resolved soon. Too many features are missing from the app, and too many employ the standard Blackboard interface without modification.

Blackboard needs to focus on making its web interface mobile friendly, by applying responsive design elements to its flagship product. At this time, instructors and students who want or need to use Blackboard on a mobile device must use a mixture of apps and a web browser to get access to the product’s major features, while some tools remain inaccessible. There are curious inconsistencies in the web interface. For instance, an instructor using Mobile Safari can edit SafeAssignments, but not regular Assignments.

Worse, the number of apps needed to use Blackboard is increasing. Over the summer, Blackboard released a new app to allow students to access Blackboard Collaborate sessions. Why couldn’t this have been integrated into the main Blackboard Mobile Learn app? At this point, to access class materials in Blackboard or integrated into Blackboard, a student at the University of Missouri-Columbia would theoretically need:

• Blackboard Mobile Learn
• A web browser
• The Blackboard Collaborate app – not yet available on Android
• The Tegrity app
• The Dropbox app

While Blackboard develops improvements and changes to their Mobile Learn application, there is little communication to institutions about where BBML will be and what it will look like in the coming two, three, or five years.  Most of the communication about the app from the company in the past has solely focused on marketing and trying to sell institutional licenses to customers.  Blackboard needs to develop and publicize a plan which gives customers a roadmap on how it will integrate the web application into the app over the coming years.  That lack of forethought might be hurting adopting and institutional purchases of BBML.

In the end, those student and instructor users who want to keep up-to-date on the activity in their courses and can afford the cost of the app will find it worthwhile.  Those who hope that Mobile Learn will be a replacement to using Blackboard on a web browser will be disappointed.  The product has found some use, but it’s lack of integration with its own tools and file structures make many in higher education wonder where the company plans to take the mobile app.

Well back to school went pretty well for myself and those of us working with Blackboard at Mizzou. However one word did rear its ugly head over the start of the Fall semester. ActiveMQ. That program could frustrate the Dahil Llama. If you are not familiar with ActiveMQ, and if you are a Blackboard admin count yourself very lucky, here’s a quick breakdown. Every application server in a Blackboard Learn instance uses ActiveMQ. If you are running multiple servers like we do at Mizzou, one of those servers is the broker. Call him the CIO or chief of the application nodes. ActiveMQ allows each server to talk to one another and the broker helps to keep the peace and make sure every server knows how to talk to the others. It’s a great tool, but earlier this year when we would run our log rotation process the servers would not restart. This created some early mornings and sleepless nights. (See An “Active” Week for the Blackboard App Admin… blog post.) We thought we had the problem licked, but it came back and in full force. Again every morning at least one server would not come back properly and sometimes it would be several servers. Our production instance would never be fully down, but service would be degraded for users. Not something you want to wake up to find every morning.

So what was the issue? We were rotating our logs on every application server one at a time with a 15 minute window. This rotation of logs required us to stop and then start Blackboard back up again. We found that when the log rotation started on the broker server, all the other servers would try to update a table in the database. Each server would try and lock the table on the database. There were more locks on that table then Mizzou football players on a fumble. We would then have to call in our Oracle DBA Team to clear the locks and allow the servers to restart.

After our first incident, we collected a lot of data for Blackboard Support. We went through all of it and created a ticket. We quickly had our issue moved up to Tier 2 and 3. After multiple back and forths with the support team. Blackboard recommended us to go back to the original peer discovery mode (see the blog post mentioned earlier). We had changed from this mode after a recommendation from Blackboard Support a few months earlier. We planned to implement the change during a maintenance window. However, our users started to experience major issue with Blackboard one evening. I requested an immediate change to apply the change and it was approved. We did see some improvement, but the issue happened again. Blackboard Support recommended the addition of a new index to the table that all the application servers were trying to update. That was applied as well.

I’m not sure if that issue is fixed or if there will be at least a triology of these posts about ActiveMQ.

The ET@MO team also brought me some interesting issues as well. The music department was trying to print out some grade reports for students. When they tried to use the grade reporting tool, the page breaks were ignored by the printer. After some digging with Blackboard Support they found a patch for this issue. We tested it in QA and applied it this weekend.

We also experienced some database troubles over the first week of classes. Our Oracle Database team found that the issue was a specific query which we reported to Blackboard. We also got a patch to fix that issue and applied it this weekend. We also applied the security patch from Blackboard this weekend as well.

We also applied patches or made configuration changes for the following issues:

• When trying to see a student’s discussion board in the Performance Dashboard, a user sometimes gets and error.
• When trying to use the Batch Import/Export tool, a warning displays in the command line but the process completes successfully.
• When editing a question in an assessment/survey and leaving it for over 20 minutes, clicking on the submit button will give the user an error.

Well I’m sure I will find much more to deal with in the coming days.

Technically Yours,

Blackboard Guru

P.S. – Much thanks to Cory, Jory, and Pamela in Blackboard Support for the help on some of the cases I mentioned.