HTTP Stream SessionID

I've created this bit of code:

For a=1 To 10
	Print GetHTTP("www.yahoo.com")
Next 

Function GetHTTP:String(sURL:String)
	Local TXT:String=""
    Local In:tStream=ReadStream("HTTP::" + sURL)
	If Not In RuntimeError "Failed to open " + sURl
	
	While Not Eof(In)
		TXT=TXT + ReadLine(In)
	Wend
    CloseStream In
	Return TXT
End Function

Which works, but each time I make a request I get a new sessionID.. How Do I maintain a session, thus keeping the same sessionID?

Okay, I've come up with a more elaborate way that I'd think would keep the same connection thus keeping the same session id, but still no luck. Also the timing is buggy. Apologize for the sloppy code, I haven't cleaned it up yet.

Local sUrl:String="www.blitzbasic.com"
Local sPage:String="faq/faq.php"

Local TXT:String
Local A:Int
Local WebSock:tSocket=CreateTCPSocket()

ConnectSocket(WebSock,HostIp(sURL),80)

While Not SocketConnected(WebSock)
' should put timeout in here then
Wend

Print "We Connected,now write out request."

Local WebStream:tSocketStream=CreateSocketStream(WebSock,False)

Print "Write out first request"

WriteLine WebStream,"GET /" + sPage +" HTTP/1.1"
WriteLine WebStream,"Host: " + surl 
WriteLine WebStream,"User-Agent: BlitzBrowser"
WriteLine WebStream,"Accept: */*"
WriteLine WebStream,""
FlushStream(WebStream)

TXT="Start"
A=MilliSecs()+10
While Len(TXT)>0 Or A>MilliSecs()
	TXT=ReadLine(WebStream)
	Print TXT
Wend
'==========================================
Print "Write out second request"

WriteLine WebStream,"GET /" + sPage +" HTTP/1.1"
WriteLine WebStream,"Host: " + surl 
WriteLine WebStream,"User-Agent: BlitzBrowser"
WriteLine WebStream,"Accept: */*"
WriteLine WebStream,""
FlushStream(WebStream)

TXT="Start"
A=MilliSecs()+1000
While Len(TXT)>1 Or A>MilliSecs()
	TXT=ReadLine(WebStream)
	Print TXT
Wend

*Edit cleaned up second example a little more.

Figured it out, you have to store the cookie it sends you and send it back each additional request. Wish the freaking HTTP 1.1 Docs said something about the cookie. Anyway, Once I clean it up I'll probably post the code. Sessions rock. What would be awsome is if there would be someway that blitz streams could do this all automatically, but I doubt that would very important to anyone but me.

Okay dealing with ReadLine was buggy, sometimes when you call it, it just sits, sometimes it returns "0", and eof for stream is never set. I think Readline should just return an empty string if nothing is available. Anyways, this code works:

Global WebSock:tSocket ' our lovely socket
Global WebStream:tSocketStream ' our lovely stream
Global TheURL:String=""
Global EndRequest:String="0"

Global SessionCookie:String="" ' cookie for saving web session

Local TXT:String,TXT2:String=""
Connect("www.yahoo.com")
Request("")

TXT = PollData(1000)
While TXT<>""
	TXT2=TXT2+ TXT
	TXT = PollData(1000)
Wend
Print TXT2
Print "Closing"

Close()
Print "Done"

	
' connect to server
	Function Connect(sURL:String,iTimeOut:Int=5000)
		TheURL=sURL
		WebSock=CreateTCPSocket()
		ConnectSocket(WebSock,HostIp(sURL),80)
		iTimeOut = iTimeOut + MilliSecs()
		While Not SocketConnected(WebSock) 
			If MilliSecs()>iTimeOut Then
				Return 0 ' failed due to timeout
			End If
		Wend
     ' We are Connected, create our streamer
		WebStream=CreateSocketStream(WebSock,False)
		Return 1 ' success
	End Function
	
' Requests data from web server
	Function Request(sPage:String)
		Local TXT:String
		Local NL:String= Chr(13) + Chr(10)
		EndRequest=""
		' Form a HTTP 1.1 Get Request
		TXT="GET /" + sPage + " HTTP/1.1" + NL
		TXT=TXT + "Host: " + TheURL + NL
		TXT=TXT + "User-Agent: BlitzBrowser" + NL
		TXT=TXT + "Accept: */*" + NL
		'TXT=TXT + "Content-Length: 0" + NL
		If SessionCookie<>"" Then
			TXT = TXT + SessionCookie + NL
		End If
		WriteLine WebStream, TXT
		FlushStream(WebStream)
	End Function
' Poll for incoming data	
	Function PollData:String(sTimeOut:Int=1)
		Local TXT:String="zx",TXT2:String=""
		If EndRequest="0" Then Return ""
		sTimeOut =sTimeOut+MilliSecs()
		While sTimeOut>MilliSecs() 
			TXT = ReadLine(WebStream)
			If String(TXT)="0" Then 
				EndRequest="0"
				Return TXT2+TXT
			End If
			If Instr(TXT,"</html>",1)>0 Then
				EndRequest="0"
				Return TXT2 + TXT
			End If
			If Len(TXT)>0 Then TXT2=TXT2 + TXT + Chr(13) + Chr(10)
			' if no cookie 
			If SessionCookie="" Then
					If Instr(TXT,"Set-Cookie",1)>0 Then
						If Not Instr(TXT,"n/a")>0 Then
							SessionCookie=Replace(TXT,"Set-Cookie","Cookie")
						End If
					End If
			End If
		Wend
		Return TXT2
	End Function
' Close Connection
	Function Close()
		SessionCookie=""
		CloseStream(WebStream)
		CloseSocket(WebSock)
	End Function

The http/1.1 protocol sends a byte count which could be used to create a bank stream buffer (read the reply into a bank) which will give you a usable eof() which raw streams obviously can't provide given 1.1 connections are left open by the server.

It's a byte count of the content, but does not include the HTTP header. So how do I know when the header ends unless I assume an <html> tag?

From memory a blank line marks the end of the header.

Then I have to assume ReadLine is returning nothing because its a new line, not sure that is safe to do. I could test it more. But sounds like a good idea.

is it not two newline characters for the end of a header?

I'm only getting one, but I think it's probably the best way to go. Get the content length from the header designated by a empty new line (readline returns nothing). Then read the content length in or hit a timeout.. I'll recode it tommorrow night after work.

I only mentioned it since it's what I do with the cgi apps I've written. And in the CGI module I wrote, PrintHTMLHeader does:

Print "Content-type:  text/html~n~n"

Well, I rewrote it to use the length. Works with my host because thier servers send Content-Length: xxx, but yahoo,google, BlitzMAx all appear to be using another method (Chunking) which I'm not interested in as I just need it to work with my site.

Global WebSock:tSocket ' our lovely socket
Global WebStream:tSocketStream ' our lovely stream
Global TheURL:String=""
Global SessionCookie:String="" ' cookie for saving web session

Local TXT:String

HTTPConnect("www.asitethatsupportscontentlength.com")
TXT=HTTPRequest("")
Print "Body--------"
Print TXT

Print "Closing"

HTTPClose()
Print "Done"

	
' connect to server
	Function HTTPConnect(sURL:String,iTimeOut:Int=5000)
		TheURL=sURL
		WebSock=CreateTCPSocket()
		ConnectSocket(WebSock,HostIp(sURL),80)
		iTimeOut = iTimeOut + MilliSecs()
		While Not SocketConnected(WebSock) 
			If MilliSecs()>iTimeOut Then
				Return 0 ' failed due to timeout
			End If
		Wend
     ' We are Connected, create our streamer
		WebStream=CreateSocketStream(WebSock,False)
		Return 1 ' success
	End Function
	
' Requests data from web server
	Function HTTPRequest:String(sPage:String,iTimeOut:Int=5000)
		Local TXT:String,A:Int
		Local tBody:String="x" ' body of html
		Local tHeader:String="x" ' header of html
		Local NL:String= Chr(13) + Chr(10)
		Local ContentLength:Int
		' Form a HTTP 1.1 Get Request
		TXT="GET /" + sPage + " HTTP/1.1" + NL
		TXT=TXT + "Host: " + TheURL + NL
		TXT=TXT + "User-Agent: BlitzBrowser" + NL
		TXT=TXT + "Accept: */*" + NL
		'TXT=TXT + "Content-Length: 0" + NL
		If SessionCookie<>"" Then
			TXT = TXT + SessionCookie 
		End If
		WriteLine WebStream, TXT
		FlushStream(WebStream)
		' now read in header grabbing length
		iTimeout=iTimeOut + MilliSecs()
		TXT=""
		Repeat 
			If MilliSecs()>iTimeOut And SocketReadAvail(WebSock)=0 Then Exit
			If SocketReadAvail(WebSock)>0 Then
				If tHeader="x" Then
					' header not complete yet
					TXT=TXT + ReadString(WebStream,SocketReadAvail(WebSock))
					' if end of header has been reached
					A=Instr(TXT,NL+NL,1)
					If A>1 Then 
						 ' grab header, remainder is body
						tHeader=Mid(TXT,1,A+2)
						tBody=Mid(TXT,Len(tHeader)+1)
						' save content length
						A=Instr(tHeader,"Content-Length:",1)
						TXT=Mid(tHeader,A+Len("Content-Length:"))
						TXT=Mid(TXT,1,Instr(TXT,NL,1))
						ContentLength=Int(TXT)
						TXT=""
						' save ASP session cookie
						If Instr(tHeader,"Set-Cookie: ASP",1)
							A=Instr(tHeader,"Set-Cookie: ASP",1)
							TXT=Mid(tHeader,A+Len("Set-Cookie:"))
							TXT=Mid(TXT,1,Instr(TXT,NL,1)-1)
							SessionCookie="Cookie:" + TXT + NL
							TXT=""
						End If
						' in case we get the whole thing in one shot check body length
						If Len(tbody)>=ContentLength Then Exit
					End If
				Else
					tBody=tBody + ReadString(WebStream,SocketReadAvail(WebSock))
					If Len(tbody)>=ContentLength Then Exit
				End If				
			End If
		Forever
		Print "Header-----"
		Print tHeader
		
		'Print "Body-------"
		'Print tBody
		Return tBody
		
	End Function
	
	' Close Connection
	Function HTTPClose()
		SessionCookie=""
		CloseStream(WebStream)
		CloseSocket(WebSock)
	End Function