Re: Max length of String - report in a single s...

Hello Sharon,

Now, after I understand your environment better, there are several
opportunities to mention. First of all we can consider your network
connection between HOST and EJB many times faster than the connection
between CLIENT (browser) and HOST. Further lets assume that the amount of
data is less than your memory heap, so we can transfer it in one virtual
packet (we fill consider optimizations to this method later).

Considering this, I would first propose a version which changes the
behaviour of your software slightly.
First I would change the interface of the EJB, so that your DTO doesn't
contain finalReport as String variable, but an attribute List<byte[]>
chunks. This way you don't have to create a String in your ejb-container
just to write the bytes.

Further I would remove the jsp from the setup (of course only if it's really
so small as your posted). In your ActionClass you could than do following:

public ActionForward execute(...){

DTO dto = EJB.getDTO();
BufferedOutputStream out = new BufferedOutputStream(res.getOutputStream
());
out.write(" <pre>\n"
.getBytes());
for (byte[] chunk:dto.getChunks()){
out.write(chunk);
}
out.write("</pre>".getBytes());
out.flush();
out.close();

}

This way you would spare at least 2 copy operations, copying into a byte
array in the stringbuffer in the ejb-container, and copying into jsp buffer.
Effectively you will spare much more on the ejb side, since the data will
probably come in in many chunks.

However, this will not save all of your problems, especially with really
LARGE data amounts. I'm not sure, how you want the browser to handle 1 GB of
data, or more, but lets assume it can handle it somehow (still it will
probably be your greatest pain).

For really large amounts of data there are various options, two of them
coming quickly into my mind:
through-streaming and polling.
Polling would mean, that in your ejb a thread exists which gets portions of
the data in the background and stores them temporarly. Than the action would
look like (some details omitted, just to give your the pic):

class ChunkWrapper{
byte[] chunk;
boolean hasNext;

boolean hasNext(){ return hasNext; }
byte[] getChunk() { return chunck; }
}

Action.execute:

String handle = EJB.startGettingInBackground();
BufferedOutputStream out = new BufferedOutputStream(res.getOutputStream
());
out.write(" <pre>\n"
.getBytes());
ChunkWrapper w;
do{
w = EJB.getNextChunk(handle);
out.write(w.getChunk());
}while(w.hasNext());
EJB.notifyHandleReadingFinished(handle);
out.write("</pre>".getBytes());
out.flush();
out.close();

This way you'd save all the memory in the HOST. However the EJB will only
have slight memory improvements, since it has to cache the complete data in
the background. I don't know how fast your CLIENT connection and how fast
your connection to the legacy system are, in case they are similar, you
could modify ejb to actually call the next chunk on the data in the
EJB.getNextChunk call and do not cache anything. However in this case your
connection to the legacy system should be able to last very long and handle
aborts in reading.

Another option is the Through-Streaming. In this case you'd need to open a
socket in the ejb, connect to this socket from your HOST and flush data
read to the output stream directly:

BufferedOutputStream out = new BufferedOutputStream(res.getOutputStream());
BufferedInputStream in = new
BufferedInputStream(howeverCreatedInputStreamFromEJBServer);
byte[] data;
out.write(" <pre>\n"
.getBytes());
while( (int read=in.read(data))!=0)
out.write(data,0,read);
out.write("</pre>".getBytes());
out.flush();
out.close();
in.close();

Please note, that I haven't compiled any of the source code, so don't expect
it to work without bugs (or proper exception handling) :-)

Regards
Leon

On 1/2/07, Sharon Jolly <Sharon.Jolly@(protected):
>
> Hi,
>
> Thank for the quick response. I do retrieve and cache the report in chunks
> before i put it into the request scope as string. But i do not have control
> over the amount of data that i would retrieve from the host for the report.
> If may be few bytes or may be 1GB, how do you think that i a StringBuilder
> or a StringBuffer would help me when it comes to JSP. If possible can you
> put down a small example.
>
> This is how my app looks like now.
>
> HOST --> EJB (here i construct a DTO with a String variable called
> finalReport)
> finalString is a String thats been toStringed from a StringBuffer.
>
> Action Class:
> request.setAttribute("printview", responseDto);
>
> JSP (get the finalString from the request object and use bean:write to
> write on the page)
> <logic:present name="printview">
> <pre><bean:write
> name="printview" property="finalReport" /></pre>
> </logic:present>
>
> I hope this is clear and could you please suggest me a good approach to
> handle any size data.
>
> Thank you.
> *
> Regards,
> Sharon Jolly
> *
> ------------------------------
> *From:* Leon Rosenberg [mailto:rosenberg.leon@(protected)]
> *Sent:* Wed 1/3/2007 12:34 AM
> *To:* Struts Users Mailing List
> *Subject:* Re: Max length of String - report in a single string for
> display
>
> What about retrieving and storing byte chunks? This way you don't have
> to copy the data and its easier to write out the chunks one-by-one for
> the outputstream too, since writing out a huge string is kindof messy
> for the performance.
> However it doesn't matter for the vm whether you store the data in
> bytes or in strings, as long as the amount is the same, if you have no
> heap you are screwed either way.
>
> Of course you could pull the legacy system and write out the data
> chunks immediately, this would remove the duty of caching them before
> writing out and reduce memory overhead, but it would be hard to do so
> if you get the data in the ejb layer (Or say less trivial).
>
> One more sidenote: if you can ensure thread-safeness you should use
> the StringBuilder instead of StringBuffer (jdk1.5 and above) since its
> faster.
>
> regards
> Leon
>
> On 1/2/07, Sharon Jolly <Sharon.Jolly@(protected):
> > Hi,
> >
> > I am currently working on a struts based web application which interacts
> with legacy systems to retrieve data for display and update. There is one
> scenario in my application where the ejb layer makes multiple host calls to
> the legacy system to retrieve data for a report. I was handling this by
> constructing a string buffer each time the host retrieves 1821 bytes of data
> and when i get an end of file flag from the host, i do a toString to the
> string buffer and set it to the request scope in my Action class for dispaly
> in the report jsp.
> >
> > Problem Definition:
> > The above scenario works fine as long as the size of the string in
> managable. Now I want to increase the size of the report in the host to
> really large one. Say previously i was displaying 1000 lines in the report
> using String, now i want to increase it to 1,00,000 lines in the report. As
> far as i know the String concept that i used previously will not work fine
> here, as max length of the String is based on the heap size in JVM. (Am not
> sure as to what's the max size of a string)
> >
> > I would really apperciate if someone could suggest me a good approach
> for this problem so that i can come up with a design that would allow me to
> display the entire report to the user irrespective of the size of the
> report, that i retrieve from the host. (Any number of lines)
> >
> > Thank you and happy new year to one and all.
> >
> > Regards,
> > Sharon Jolly
> >
> >
> ----------------------------------------------------------------DISCLAIMER---------------------------------------------------------
> > Information transmitted by this EMAIL is proprietary to iGATE Group of
> Companies and is intended for use only by the individual
> > or entity to whom it is addressed and may contain information that is
> privileged, confidential, or exempt from disclosure under
> > applicable law. If you are not the intended recipient of this EMAIL
> immediately notify the sender at iGATE or mailadmin@(protected)
> > and delete this EMAIL including any attachments
> >
> >
>
>
>
>
> ----------------------------------------------------------------DISCLAIMER---------------------------------------------------------
>
> Information transmitted by this EMAIL is proprietary to iGATE Group of Companies and is intended for use only by the individual
>
> or entity to whom it is addressed and may contain information that is privileged, confidential, or exempt from disclosure under
>
> applicable law. If you are not the intended recipient of this EMAIL immediately notify the sender at iGATE or
> mailadmin@(protected)
> and delete this EMAIL including any attachments
>

Re: Max length of String - report in a single string for display

Leon Rosenberg

2007-01-02