Search Mailing List Archives


Limit search to: Subject & Body Subject Author
Sort by: Reverse Sort
Limit to: All This Week Last Week This Month Last Month
Select Date Range     through    

[java-nlp-user] Fw: JVM heap size problem

John Bauer horatio at gmail.com
Sun Apr 24 13:35:32 PDT 2011


You need to serialize the Annotation, not the StanfordCoreNLP ;)

The Annotation contains all of the results of the process.  The
pipeline itself doesn't contain any information,

John

On Sun, Apr 24, 2011 at 7:08 AM, Ayah Magdy <ayah_elmaghraby at yahoo.com> wrote:
> This is the code in the main function...
> Properties props1 = new Properties();
>    props1.put("annotators", "tokenize, ssplit, pos, lemma, parse");;
>
>    StanfordCoreNLP pipeline1 = new StanfordCoreNLP(props1);
>    // read some text in the text variable
>    String text = "Stanford University is located in California. It is a
> great university.";
>
>    // create an empty Annotation just with the given text
>    Annotation document = new Annotation(text);
>
>    // run all Annotators on this text
>    pipeline1.annotate(document);
>
>    pipeline1.PoolCopy();
> // Write to disk with FileOutputStream
>   FileOutputStream f_out = new
>   FileOutputStream("myobject.data");
>   // Write object with ObjectOutputStream
>   ObjectOutputStream obj_out = new
>   ObjectOutputStream (f_out);
>   // Write object out to disk
>   obj_out.writeObject (pipeline1);
> I added a parameter copyPoool to the StanfordCoreNLP class to save a copy of
> the static pool on it as i didnt want the pool to register the already done
> stages in the pipeline and to start from ner stage in the next stage...
> but the program throws saying CoreLabel$2 isn't serializable ... i added
> "implements serializable" to some class like parserData & LexerParser and
> couple more classes until i reached this error.... i also made the parameter
> "lexer " in class Morphology transient to avoid the error of the
> StringReader not being serializable
> ________________________________
> From: John Bauer <horatio at gmail.com>
> To: Ayah Magdy <ayah_elmaghraby at yahoo.com>
> Cc: java-nlp-user at lists.stanford.edu
> Sent: Sun, April 24, 2011 1:09:34 AM
> Subject: Re: [java-nlp-user] Fw: JVM heap size problem
>
> Please send a code snippet...
>
> John
>
> On Sat, Apr 23, 2011 at 1:29 PM, Ayah Magdy <ayah_elmaghraby at yahoo.com>
> wrote:
>>
>> While trying to serialize the CoreStanfordNLP i got an error telling me
>> CoreLabel$2 isnt serializable i couldnt find this class all i can find is
>> class corelabel. how do i fix this error?
>> ________________________________
>> From: John Bauer <horatio at gmail.com>
>> To: Ayah Magdy <ayah_elmaghraby at yahoo.com>
>> Sent: Fri, April 22, 2011 10:03:19 PM
>> Subject: Re: [java-nlp-user] JVM heap size problem
>>
>> It's not supported any more,
>>
>> John
>>
>> On Apr 22, 2011 10:05 AM, "Ayah Magdy" <ayah_elmaghraby at yahoo.com> wrote:
>>> i meant the "coref" annotator as i found it in the documentation but when
>>> i used
>>> it it didnt work perhaps i was using it wrong
>>>
>>> ________________________________
>>> From: John Bauer <horatio at gmail.com>
>>> To: Ayah Magdy <ayah_elmaghraby at yahoo.com>
>>> Cc: java-nlp-user at lists.stanford.edu
>>> Sent: Fri, April 22, 2011 6:53:23 PM
>>> Subject: Re: [java-nlp-user] JVM heap size problem
>>>
>>>> To do this how can i find the start point of each annotation class... i
>>>> tried looking in the code of the pipeline to find who calls what but i
>>>> got
>>>> lost...
>>>> where does every annotation start or where can i find it?
>>>
>>> That's not what I meant. What I meant was run three separate
>>> pipelines. First run with the annotators "tokenize, ssplit, pos,
>>> lemma, parse". Serialize this to disk. Then, in a second program,
>>> load it back in and run with just the annotator "ner". Serialize to
>>> disk again. Then, in a third program, load it back in and run with
>>> just the annotator "dcoref".
>>>
>>> Honestly, you might still run out of memory depending on document
>>> size, but it might work.
>>>
>>>> how can i use the coref annotation? is it available anymore?
>>>
>>> It will be used if you include "dcoref" in your list of annotators.
>>> What makes you think it isn't there any more?
>>>
>>>> what is the properties taken by this class in its constructor?
>>>
>>> I suppose they aren't well documented. You can look in
>>> src/edu/stanford/nlp/dcoref/Constants.java to see what properties are
>>> available, and you can hunt through the code to see what effect they
>>> have... we'll add more documentation at a future date.
>>>
>>> Jon
>>>
>>> On Fri, Apr 22, 2011 at 7:09 AM, Ayah Magdy <ayah_elmaghraby at yahoo.com>
>>> wrote:
>>>> To do this how can i find the start point of each annotation class... i
>>>> tried looking in the code of the pipeline to find who calls what but i
>>>> got
>>>> lost...
>>>> where does every annotation start or where can i find it?
>>>> ________________________________
>>>> From: John Bauer <horatio at gmail.com>
>>>> To: Ayah Magdy <ayah_elmaghraby at yahoo.com>
>>>> Cc: java-nlp-user at lists.stanford.edu
>>>> Sent: Fri, April 15, 2011 6:35:40 PM
>>>> Subject: Re: [java-nlp-user] JVM heap size problem
>>>>
>>>> Coref needs each of the other annotators, so you're basically hosed.
>>>>
>>>> One thing you can do is run some of the annotators and serialize the
>>>> output. Then you would start a second program which reads in the
>>>> serialized output and runs the other annotators. We don't have any
>>>> support for that, but it shouldn't be too hard to do.
>>>>
>>>> John
>>>>
>>>> On Fri, Apr 15, 2011 at 9:24 AM, Ayah Magdy <ayah_elmaghraby at yahoo.com>
>>>> wrote:
>>>>> I am working on windows 7 32 bit system, eclispe & Jrockit i can hardly
>>>>> get
>>>>> 1.7G heap size i tried increasing virtual memory size.. nothing worked
>>>>> to
>>>>> get a 1.8G memory needed to run the corenlp...
>>>>> all i need from the core is the coreference resolution so how can i get
>>>>> it
>>>>> without running the whole pipeline , is there a solution around this
>>>>> problem
>>>>> ?
>>>>> _______________________________________________
>>>>> java-nlp-user mailing list
>>>>> java-nlp-user at lists.stanford.edu
>>>>> https://mailman.stanford.edu/mailman/listinfo/java-nlp-user
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> java-nlp-user mailing list
>>>> java-nlp-user at lists.stanford.edu
>>>> https://mailman.stanford.edu/mailman/listinfo/java-nlp-user
>>>>
>>>>
>>
>> _______________________________________________
>> java-nlp-user mailing list
>> java-nlp-user at lists.stanford.edu
>> https://mailman.stanford.edu/mailman/listinfo/java-nlp-user
>>
>>
>
> _______________________________________________
> java-nlp-user mailing list
> java-nlp-user at lists.stanford.edu
> https://mailman.stanford.edu/mailman/listinfo/java-nlp-user
>
>



More information about the java-nlp-user mailing list