Search Mailing List Archives


Limit search to: Subject & Body Subject Author
Sort by: Reverse Sort
Limit to: All This Week Last Week This Month Last Month
Select Date Range     through    

[bioontology-support] anotator

John Graybeal jgraybeal at stanford.edu
Wed Nov 15 11:27:53 PST 2017


could u point me to the actual document/paper/reference with this evaluation?

I can't, because I didn't capture that particular reference and it would take me too long to rediscover it.

A quick search on Google ("biomedical annotations comparison ncbo") yields a number of possible comparisons, Google scholar might narrow the field further. In particular this link is tolerably recent and tolerably complete: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0116040

And the proceedings paper that Clement previously referenced on this list can be found at http://ceur-ws.org/Vol-1866/paper_62.pdf; it contains a number of references to previous evaluations.

John


On Nov 15, 2017, at 10:38 AM, Alexander Garcia Castro <alexgarciac at gmail.com<mailto:alexgarciac at gmail.com>> wrote:

Hi John, "In general, the only technical evaluation I've seen has shown BioPortal first or second (if I recall correctly) in terms of speed and completeness of annotation. " could u point me to the actual document/paper/reference with this evaluation?



On Sat, Sep 30, 2017 at 12:32 AM, John Graybeal <jgraybeal at stanford.edu<mailto:jgraybeal at stanford.edu>> wrote:
Hi Alexander,

A quick answer, sorry I don't have all the research data at hand but perhaps you can use Google Scholar to find more precise information if you need it.

Since we consider BioPortal to be the biggest source of Biomedical Ontologies, we definitely consider our annotator the best service for annotations against a large and diverse number of ontologies. (And therefore, also the most reliable service in that category. :->)

The article "Semantic annotation in biomedicine: the current landscape"  in Journal of Biomedical Semantics (https://jbiomedsem.biomedcentral.com/articles/10.1186/s13326-017-0153-x) has a survey of many, if not all, of the existing semantic annotation technologies. NCBO Bioportal appears in Tables 1 and 3 (scroll the table left on screen). While quite dated in some respects (BioPortal has over 500 BioMedical ontologies, and it is open source, available on GitHub) there is a lot of useful information here. I know there are some pipeline annotators that leverage BioPortal services to provide their own annotation capabilities, but can not offer the details off-hand, sorry.

In general, the only technical evaluation I've seen has shown BioPortal first or second (if I recall correctly) in terms of speed and completeness of annotation. Of course, without specific evaluation criteria and comparisons, it is all but impossible to have a meaningful rebuttal to the observation "something else is better". Certainly there are features available with some other annotators that Bioportal does not provide, so there can be tradeoffs depending on specific needs. Conversely, among many other features, BioPortal supports both free text (paragraphs of content), and the ability to annotate the largest matching strings. These often are not available in other annotators.

With regard to reliability, the most common concern raised about BioPortal is that it is not consistently responsive, especially for people with long-running and large-scale annotation processes and similar complex queries. Our monitoring services show the annotation service is up over 99.5% of the time, but it is definitely challenged by repeatedly annotating pages or more of text against all 500+ ontologies 15 times per second (per requestor), whether your text, or someone else's. We regularly get requests for the Virtual Appliance so that people can set up their own annotation pipeline, with only the ontologies they are interested in, and handling queries as fast as they can send them. That is an example of a more reliable service, but not a public one.

Perhaps this information will be sufficient to help you, or will bring forward other comments from the user list.

john


On Sep 27, 2017, at 6:59 AM, Alexander Garcia Castro <alexgarciac at gmail.com<mailto:alexgarciac at gmail.com>> wrote:

Hi all. I am using the NCBO annotator and I like it. In a paper I just submitted the reviewer is saying that there are other NER tools that are better. this may be true but when processing the whole of PMC over a web service against all biomedical ontologies or a big portion of them then... is there anything as reliable as bioportal annotator? could some one help me out with some reference that justifies the choice of  the ncbo annotator? I am using it because I dont know if there is a web service that is as reliable when processing lots of texts and also that works against all of biomedical ontologies -in my case it is pretty much a choice due to the need to reuse other peoples infrastructure because I would not be able to process it on my own.

also, there are lots of other NER tools but why was this one specifically selected for the NCBO NER service? as oppose to using another NER tool for the NCBO annotator.

cheers.

--
Alexander Garcia
https://www.researchgate.net/profile/Alexander_Garcia
http://www.usefilm.com/photographer/75943.html
http://www.linkedin.com/in/alexgarciac

_______________________________________________
bioontology-support mailing list
bioontology-support at lists.stanford.edu<mailto:bioontology-support at lists.stanford.edu>
https://mailman.stanford.edu/mailman/listinfo/bioontology-support

========================
John Graybeal
Technical Program Manager
Center for Expanded Data Annotation and Retrieval /+/ NCBO BioPortal
Stanford Center for Biomedical Informatics Research
650-736-1632





--
Alexander Garcia
https://www.researchgate.net/profile/Alexander_Garcia
http://www.usefilm.com/photographer/75943.html
http://www.linkedin.com/in/alexgarciac


========================
John Graybeal
Technical Program Manager
Center for Expanded Data Annotation and Retrieval /+/ NCBO BioPortal
Stanford Center for Biomedical Informatics Research
650-736-1632


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.stanford.edu/pipermail/bioontology-support/attachments/20171115/7f02aaf2/attachment-0001.html>


More information about the bioontology-support mailing list