Search Mailing List Archives
[bioontology-support] recent BioPortal outages: believed in hand
jgraybeal at stanford.edu
Wed Mar 15 17:07:52 PDT 2017
As several of you have noted (especially this morning!), we have had a lot of outages on BioPortal over the last two weeks (particularly on the API), and also some indexing and submission difficulties.
The Too Long; Didn't Read version: We have now taken actions which should return our reliability to the normally high (>99.5% availability) levels -- and let us quickly resolve other recent issues.
First, let me thank all of you who sent in reports. They are very helpful in getting our attention, and giving a sense of the type of work affected. I'm sorry we can't have 24x7 support to respond in real time, but we'll keep working on our response times and response actions, and coming up with creative solutions.
As to the problems, it appears some particular long-running queries strongly excited a bug in the system which randomly returns null data. This bug may get triggered once in a while anyway; but with heavy loads, particularly of one query type, it is triggered frequently enough to cause system failures. A lot of the problematic queries were running overnight these two weeks, hence the frequent middle-of-the-night outages.
We've sorted out the nature of the queries, and are working with the originator to move them to another server, which should minimize downtime from now on. With this in hand, soon we should be able to complete re-indexing of all our ontologies, and restart auto-processing of new submissions.
We have already identified several maintenance strategies that should localize further issues, and give us a better opportunity to find the root cause of this issue (that is, the original bug). These activities will go on behind the scenes, and should not affect system operations in the meantime. Having this understanding should help us significantly improve the future stability of the system as well.
Thank you for your patience and your communications. We welcome further input at any time.
Technical Program Manager
Center for Expanded Data Annotation and Retrieval /+/ NCBO BioPortal
Stanford Center for Biomedical Informatics Research
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the bioontology-support