Search Mailing List Archives


Limit search to: Subject & Body Subject Author
Sort by: Reverse Sort
Limit to: All This Week Last Week This Month Last Month
Select Date Range     through    

[protege-discussion] RMI server scalability for Protege Frames 3.5 editor

Timothy Redmond tredmond at stanford.edu
Mon Jul 8 09:15:16 PDT 2013


On 07/02/2013 04:24 PM, John Pierre wrote:
> Thank you very much for your help Timothy, the tgviz tab was indeed 
> the problem.  I'm glad there was such a simple explanation, luckily we 
> aren't even using that plugin.
>
> Let me know if there are some examples for the proper style to code 
> plugins for the client-server architecture, we may be developing a 
> plugin in the near future.

The Protege 3 Server works by running a fine grained knowledge base 
interface over a network.  In particular the interface in question is 
very similar to the frame store interface.   A big problem with this 
type of design is that it is very easy and natural to write code that 
when executed results in a large number of calls across the network.

So a strategy that works is to make several calls to ask for various 
properties of a frame such as a class or an individual.  The reason that 
this works is that the server can anticipate future requests from the 
client and generate a cache of data that the client will need in future 
calls.

A strategy that doesn't work is to loop through all the frames in the 
ontology and successively ask for information about each of these 
frames.  There is no way to build a cache in this scenario without 
essentially loading the entire ontology on the client.  I suspect that 
the tgviz tab was doing something like this.

If you need to gather this information, the trick is to use a 
ProtegeJob.  A protege job will be sent to the server and will execute 
on the server if the Protege client is connected to a server.  This 
allows a complex operation to happen in server space without requiring a 
large amount of individual calls across a network.

-Timothy


>
>
>
>
>
> On Fri, Jun 28, 2013 at 2:37 PM, Timothy Redmond 
> <tredmond at stanford.edu <mailto:tredmond at stanford.edu>> wrote:
>
>
>     Sorry for the delay in the response.  The problem that you are
>     having appears to occur in the TGViz tab.  When the tab is
>     disabled, the protege client comes up quickly (over here at least)
>     even when I simulate fairly extreme network delays (800 kilobits
>     per second and 80 milliseconds of latency).  But when the TGViz
>     tab is enabled, it takes a long time to come up.  I have almost
>     finished writing this message and the TGViz tab is still trying to
>     initialize.
>
>     This is an old plugin and it hasn't been rewritten to accommodate
>     the protege client-server architecture. Unfortunately the Protege
>     3 client-server architecture requires that plugin code be written
>     in a certain style to avoid major slow-downs of this type.
>
>     You should be able to fix the problem by shutting down the server,
>     loading the project used by the server and disabling the tgviz tab
>     and then restarting the server. Alternatively you can test this
>     theory with a client where the tgviz tab has been removed.
>
>     -Timothy.
>
>
>
>     On 06/23/2013 02:57 PM, John Pierre wrote:
>>     Thanks for the thoughtful reply.  I am using Protege 3.5 Build
>>     663.   The .pprj file is only about 52K with about 150 or so
>>     class definitions for top level classes, slots, layout
>>     properties, etc.
>>
>>     Looking at the fine grained logging on the server and the client
>>     it seems it is loading 80MB of data when the client opens the
>>     ontology.  Seems to be loading all the frames into the cache with
>>     47,952 calls like this:
>>
>>     2013.06.22 12 <tel:2013.06.22%2012>:21:57.975 PDT FINE:
>>     Invocation took 3 ms -- RemoteClientFrameStore$2.invoke()
>>     2013.06.22 12 <tel:2013.06.22%2012>:21:57.975 PDT FINE: Cache
>>     miss for frame named 1125R [171078] -- RemoteClientFrameStore.ge
>>     tFrame()
>>     2013.06.22 12 <tel:2013.06.22%2012>:21:57.975 PDT FINE: Remote
>>     invoke: getFrame Args: -- RemoteClientFrameStore$2.invoke()
>>
>>     I noticed that -Dserver.client.preload.skip and
>>     -Dpreload.frame.limit properties do not seem to have any affect
>>     whatsoever.
>>
>>     I'll send some additional info off-line.  Sincere thanks!
>>
>>
>>
>>
>>
>>
>>
>>
>>     On Sat, Jun 22, 2013 at 12:49 PM, Timothy Redmond
>>     <tredmond at stanford.edu <mailto:tredmond at stanford.edu>> wrote:
>>
>>
>>         This type of performance problem is a tricky area and to
>>         understand it better we would have to understand exactly
>>         where the bottleneck is.  But my first questions would be how
>>         large is the pprj file and what version of Protege are you
>>         using.  We made a change to Protege recently that optimizes
>>         how the .pprj part of a frames project is loaded by the
>>         client.  Using the very latest version of Protege may
>>         dramatically improve the startup performance if the pprj file
>>         is large.
>>
>>         If moving to the latest Protege is not sufficient then you
>>         could send me a copy of the ontology so that I could see the
>>         performance issue and find the bottleneck here.  If you send
>>         the ontology out of line I can keep it private if needed.
>>
>>
>>>
>>>         Therefore the culprit seems to be network demands of the rmi
>>>         server.
>>
>>         This sounds like a simple
>>
>>
>>>
>>>         1. What is the practical scalability limit of the Protege
>>>         RMI client-server in terms of ontology size?
>>
>>         The server-client is used with ontologies of over 128K
>>         classes. In this example the ontology is also a mysql
>>         database project.  So the 25000 frames you describe is not
>>         that large.
>>
>>
>>>
>>>         2.  Are there additional configuration settings that we can
>>>         try to get our ontology to load?
>>
>>         It has been a long time since I worked on the protege 3
>>         client server but I think you have found the main options. 
>>         But you should also be warned that
>>
>>                      -Dserver.client.preload.skip=true
>>
>>
>>         might work against you.  You save time on the preload but all
>>         the operations thereafter are slower until the client side
>>         cache has been sufficiently built up.
>>
>>
>>>
>>>         3.  Are there other collaboration models we could try for
>>>         allowing multiple people to work on a large scale Frames
>>>         ontology besides the rmi client-server approach?
>>
>>         You could also try webprotege.
>>
>>         -Timothy
>>
>>
>>
>>
>>         On 06/19/2013 10:26 AM, John Pierre wrote:
>>>         We are trying to set up a collaborative installation so
>>>         multiple developers can edit a Frames ontology.  The
>>>         ontology currently has about 25,000 frames.
>>>
>>>         We have successfully configured the RMI server and can
>>>         connect clients to the server and access the example
>>>         ontologies across the network.
>>>
>>>         The problem is that our 25,000 frame ontology isn't able to
>>>         load into the clients when accessed across the network.
>>>
>>>         Our ontology is MySQL backed.
>>>
>>>         The ontology loads fine when accessed on the same machine
>>>         without going through the rmi server.
>>>
>>>         The ontology loads but takes several minutes to do so when
>>>         running the client and server on the same machine through
>>>         the localhost loopback.
>>>
>>>         The ontology loading hangs and cannot load when running the
>>>         client and server on different machines either on a wide
>>>         area network (20+Mbps) nor on a local area Ethernet network
>>>         (100+Mbps).  After waiting 30 mins or so we get broken pipes
>>>         and/or timed out connections.
>>>
>>>         The ontology loads if the MySQL database is accessed across
>>>         the network directly without using the client-server (.pprj
>>>         file is on the client side but points to a MySQL database
>>>         hosted on the network)
>>>
>>>         Therefore the culprit seems to be network demands of the rmi
>>>         server.
>>>
>>>         We have -Dserver.use.compression=true turned on at the server.
>>>
>>>         We've tried -Dserver.client.preload.skip=true on the client
>>>         side.
>>>
>>>
>>>         It seems this 25,000 frame ontology might be too large for
>>>         the RMI client-server architecture, but the Protege
>>>         documentation seems to hint that much larger ontologies have
>>>         been developed using Protege.
>>>
>>>         Questions:
>>>
>>>         1. What is the practical scalability limit of the Protege
>>>         RMI client-server in terms of ontology size?
>>>
>>>         2.  Are there additional configuration settings that we can
>>>         try to get our ontology to load?
>>>
>>>         3.  Are there other collaboration models we could try for
>>>         allowing multiple people to work on a large scale Frames
>>>         ontology besides the rmi client-server approach?
>>>
>>>         Thanks in advance for your help.
>>>
>>>         John
>>>
>>>
>>>
>>>
>>>         _______________________________________________
>>>         protege-discussion mailing list
>>>         protege-discussion at lists.stanford.edu  <mailto:protege-discussion at lists.stanford.edu>
>>>         https://mailman.stanford.edu/mailman/listinfo/protege-discussion
>>>
>>>         Instructions for unsubscribing:http://protege.stanford.edu/doc/faq.html#01a.03
>>
>>
>>         _______________________________________________
>>         protege-discussion mailing list
>>         protege-discussion at lists.stanford.edu
>>         <mailto:protege-discussion at lists.stanford.edu>
>>         https://mailman.stanford.edu/mailman/listinfo/protege-discussion
>>
>>         Instructions for unsubscribing:
>>         http://protege.stanford.edu/doc/faq.html#01a.03
>>
>>
>>
>>
>>     On Sat, Jun 22, 2013 at 12:49 PM, Timothy Redmond
>>     <tredmond at stanford.edu <mailto:tredmond at stanford.edu>> wrote:
>>
>>
>>         This type of performance problem is a tricky area and to
>>         understand it better we would have to understand exactly
>>         where the bottleneck is.  But my first questions would be how
>>         large is the pprj file and what version of Protege are you
>>         using.  We made a change to Protege recently that optimizes
>>         how the .pprj part of a frames project is loaded by the
>>         client.  Using the very latest version of Protege may
>>         dramatically improve the startup performance if the pprj file
>>         is large.
>>
>>         If moving to the latest Protege is not sufficient then you
>>         could send me a copy of the ontology so that I could see the
>>         performance issue and find the bottleneck here.  If you send
>>         the ontology out of line I can keep it private if needed.
>>
>>
>>>
>>>         Therefore the culprit seems to be network demands of the rmi
>>>         server.
>>
>>         This sounds like a simple
>>
>>
>>>
>>>         1. What is the practical scalability limit of the Protege
>>>         RMI client-server in terms of ontology size?
>>
>>         The server-client is used with ontologies of over 128K
>>         classes.  In this example the ontology is also a mysql
>>         database project.  So the 25000 frames you describe is not
>>         that large.
>>
>>
>>>
>>>         2.  Are there additional configuration settings that we can
>>>         try to get our ontology to load?
>>
>>         It has been a long time since I worked on the protege 3
>>         client server but I think you have found the main options. 
>>         But you should also be warned that
>>
>>                      -Dserver.client.preload.skip=true
>>
>>
>>         might work against you.  You save time on the preload but all
>>         the operations thereafter are slower until the client side
>>         cache has been sufficiently built up.
>>
>>
>>>
>>>         3.  Are there other collaboration models we could try for
>>>         allowing multiple people to work on a large scale Frames
>>>         ontology besides the rmi client-server approach?
>>
>>         You could also try webprotege.
>>
>>         -Timothy
>>
>>
>>
>>
>>         On 06/19/2013 10:26 AM, John Pierre wrote:
>>>         We are trying to set up a collaborative installation so
>>>         multiple developers can edit a Frames ontology.  The
>>>         ontology currently has about 25,000 frames.
>>>
>>>         We have successfully configured the RMI server and can
>>>         connect clients to the server and access the example
>>>         ontologies across the network.
>>>
>>>         The problem is that our 25,000 frame ontology isn't able to
>>>         load into the clients when accessed across the network.
>>>
>>>         Our ontology is MySQL backed.
>>>
>>>         The ontology loads fine when accessed on the same machine
>>>         without going through the rmi server.
>>>
>>>         The ontology loads but takes several minutes to do so when
>>>         running the client and server on the same machine through
>>>         the localhost loopback.
>>>
>>>         The ontology loading hangs and cannot load when running the
>>>         client and server on different machines either on a wide
>>>         area network (20+Mbps) nor on a local area Ethernet network
>>>         (100+Mbps). After waiting 30 mins or so we get broken pipes
>>>         and/or timed out connections.
>>>
>>>         The ontology loads if the MySQL database is accessed across
>>>         the network directly without using the client-server (.pprj
>>>         file is on the client side but points to a MySQL database
>>>         hosted on the network)
>>>
>>>         Therefore the culprit seems to be network demands of the rmi
>>>         server.
>>>
>>>         We have -Dserver.use.compression=true turned on at the server.
>>>
>>>         We've tried -Dserver.client.preload.skip=true on the client
>>>         side.
>>>
>>>
>>>         It seems this 25,000 frame ontology might be too large for
>>>         the RMI client-server architecture, but the Protege
>>>         documentation seems to hint that much larger ontologies have
>>>         been developed using Protege.
>>>
>>>         Questions:
>>>
>>>         1. What is the practical scalability limit of the Protege
>>>         RMI client-server in terms of ontology size?
>>>
>>>         2.  Are there additional configuration settings that we can
>>>         try to get our ontology to load?
>>>
>>>         3.  Are there other collaboration models we could try for
>>>         allowing multiple people to work on a large scale Frames
>>>         ontology besides the rmi client-server approach?
>>>
>>>         Thanks in advance for your help.
>>>
>>>         John
>>>
>>>
>>>
>>>
>>>         _______________________________________________
>>>         protege-discussion mailing list
>>>         protege-discussion at lists.stanford.edu  <mailto:protege-discussion at lists.stanford.edu>
>>>         https://mailman.stanford.edu/mailman/listinfo/protege-discussion
>>>
>>>         Instructions for unsubscribing:http://protege.stanford.edu/doc/faq.html#01a.03
>>
>>
>>         _______________________________________________
>>         protege-discussion mailing list
>>         protege-discussion at lists.stanford.edu
>>         <mailto:protege-discussion at lists.stanford.edu>
>>         https://mailman.stanford.edu/mailman/listinfo/protege-discussion
>>
>>         Instructions for unsubscribing:
>>         http://protege.stanford.edu/doc/faq.html#01a.03
>>
>>
>>
>>
>>     _______________________________________________
>>     protege-discussion mailing list
>>     protege-discussion at lists.stanford.edu  <mailto:protege-discussion at lists.stanford.edu>
>>     https://mailman.stanford.edu/mailman/listinfo/protege-discussion
>>
>>     Instructions for unsubscribing:http://protege.stanford.edu/doc/faq.html#01a.03
>
>
>     _______________________________________________
>     protege-discussion mailing list
>     protege-discussion at lists.stanford.edu
>     <mailto:protege-discussion at lists.stanford.edu>
>     https://mailman.stanford.edu/mailman/listinfo/protege-discussion
>
>     Instructions for unsubscribing:
>     http://protege.stanford.edu/doc/faq.html#01a.03
>
>
>
>
> _______________________________________________
> protege-discussion mailing list
> protege-discussion at lists.stanford.edu
> https://mailman.stanford.edu/mailman/listinfo/protege-discussion
>
> Instructions for unsubscribing: http://protege.stanford.edu/doc/faq.html#01a.03

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.stanford.edu/pipermail/protege-discussion/attachments/20130708/dc2d25e4/attachment-0001.html>


More information about the protege-discussion mailing list