reverse engineering the network.download

[noriega]

Hello,

For the past few days i’ve been stuck on breaking down firebase database streaming api and trying to reverse engineer the network.request (specifically for download) based on this example git

https://github.com/vsergeyev/corona-firebase/blob/master/corona_firebase.lua

From that example, though it works, I would not want to loop through the temporary file created… and corona’s api does not output any sort of response data in “progress” when I print/print_r’ed it…

So, I figured if it is possible to use a streaming api with corona’s network.request/download, I could, correct me if i’m wrong, get the response while before it it written to the temp file is somewhere in LUA that Corona’s api isn’t spitting out.  The problem is I haven’t found the right protocol to use. 

Here are some excerpts i’ve tried but does not work.

i’ve tried this but doesn’t work

local connection = http.request({ url = url, sink = ltn12.sink.table(response), redirect = false, headers = { ["accept"] = "text/event-stream", ["content-type"] = "text/event-stream", ["cache-control"] = "no-cache", ["connection"] = "keep-alive" }, })

i’ve also tried this but it just throws off closed

local connection = socket.protect(function(event) local client = socket.try(socket.connect(url, 443)) local try = socket.newtry(function(event) Runtime:removeEventListener("enterFrame", enterFrame) client:close() end) try(client:setoption("tcp-nodelay", true)) try(client:setoption("keepalive", true)) local source = try(socket.source("until-closed", client)) local function enterFrame() local input,output = socket.select({ client }, nil, 0) print(input, output) end Runtime:addEventListener("enterFrame", enterFrame)

 local client = socket.tcp() client:connect(url , 443, 0) client:setoption("tcp-nodelay", true) client:settimeout(0) client:setoption("keepalive", true) local input,output = socket.select({ client }, nil, 0) timer.performWithDelay( 200, function() local input,output = socket.select({ client }, nil, 0) for i,v in ipairs(input) do local got\_something\_new = false while true do local skt, e, p = v:receive(1) if (skt) then buffer = buffer.. skt got\_something\_new = true end if (p) then buffer = buffer..p got\_something\_new = true end if (not skt or e) then break end end -- /while-do end end, 0)

[noriega]

local socket = require('socket') local http = require('socket.http') local copas = require("copas") local ltn12 = require("ltn12") local asynchttp = require("copas.http").request local response local url = "https://firebaseurl" print("Requesting secure URL on HTTPS socket: " .. url) local master, err copas.addthread(function() print('thread') http.TIMEOUT = 0.01 master, err = asynchttp({ url = url, redirect = true, protocol = "any", options = "all", verify = "none", sink = ltn12.sink.table(response), headers = { "GET / HTTP/1.1\r\n", ["accept"] = "text/event-stream", ["cache-control"] = "no-cache", ["connection"] = "keep-alive" }, create = function(...) print(...) response = ... end }) print(master) print(err) print('yofff') end, url) copas.loop() local stored = {} local function enterFrame() local input,output = socket.select({ master }, nil, 0) if(response ~= stored) then stored = response print(stored) end if(input and #input \> 0) then print(input) end if(output and #output \> 0) then print(output) end end Runtime:addEventListener("enterFrame", enterFrame)

the latest iteration of my attempts.  still to no avail.  i’m also not sure if someone could move it to the network forum. i’m still pretty lost in trying to at least capture the partial table while it is listening for the event stream.

[rob]

The network.download()/.request() API’s should give you progress in your callback function. But generally the amount being downloaded needs to be known. The server has to convey this information to you before we can provide it.

Depending on the size of the dataset you’re downloading and if Firebase outputs JSON or not, you could just use network.request() and convert JSON results to a Lua table. No need to have a temporary file. Also if the temporary file is JSON then you can just JSON decode the file to get a Lua table.

Rob

[noriega]

The problem with the network.request/download, is that with the text/event-stream, the connection needs to be kept alive till it is cancelled and/or closed. The dataset is not a static number and is essentially always downloading/listening.  From my research, it’s like a websocket but it’s a one way connection after the initial call.  

network.download I figured uses a socket where it uses ltn12.sink.file to store the temp file as it downloads, then renames it to the appropriate filename defined in the params of params.response.filename   

i wanted to basically build the network.download but instead of saving the file to the ltn12.file, I wanted to save it to a ltn12.table 

[noriega]

local function getListener(event) if ( event.isError ) then print( "Network error: ", event.response ) else --outputs users.json file print (event.response) end end local function streamListener(event) if ( event.isError ) then print( "Network error: ", event.response ) else --workaround to see latest users.json file (called on each keep-alive, put, patch, auth\_revoked, and cancel) since we don't know the the event type                 network.request("https://project-xxxxx.firebaseio.com/users.json", "GET", getListener ) print (event.response) --is nil since not finished "downloading" end end local headers = {} headers["Accept"] = "text/event-stream" headers["Cache-Control"] = "no-cache" headers["Connection"] = "keep-alive" local params = {} params.headers = headers params.progress = "download" params.timeout = -1 params.handleRedirects = true network.request("https://project-xxxxx.firebaseio.com/users.json?sse=true", "GET", streamListener, params )

so i was able to receive events from firebase using a combination of the headers and putting a parameter called ?sse=true but I have nothing to reference from what triggered it.  (it can be keep alive, put, patch, auth_revoked, or cancel)

How hard would it be for the engineers to just to add the ability to see the the partial response within the “progress” phase?

event.response is nil until the get request is complete, but it will never be complete unless the stream is closed. (which we do not want)

this is an output of printing the (event) table on the streamListener function

table: 0x7fb02051b6f0 { utilities(285): [responseHeaders] =\> table: 0x7fb02051b6f0 { utilities(289): [Access-Control-Allow-Origin] =\> "\*" utilities(289): [Content-Type] =\> "text/event-stream" utilities(289): [Cache-Control] =\> "no-cache" utilities(287): } utilities(289): [phase] =\> "progress" utilities(291): [bytesEstimated] =\> -1 utilities(289): [name] =\> "networkRequest" utilities(291): [bytesTransferred] =\> 247 utilities(291): [status] =\> 200 utilities(289): [url] =\> "https://project-xxxx.firebaseio.com/users.json?sse=true" utilities(291): [isError] =\> false utilities(291): [requestId] =\> userdata: 0x7fb02051b788 utilities(302): }

https://firebase.google.com/docs/reference/rest/database/#section-streaming

event: event name data: JSON encoded data payload

[noriega]

local socket = require('socket') local http = require('socket.http') local copas = require("copas") local ltn12 = require("ltn12") local asynchttp = require("copas.http").request local response local url = "https://firebaseurl" print("Requesting secure URL on HTTPS socket: " .. url) local master, err copas.addthread(function() print('thread') http.TIMEOUT = 0.01 master, err = asynchttp({ url = url, redirect = true, protocol = "any", options = "all", verify = "none", sink = ltn12.sink.table(response), headers = { "GET / HTTP/1.1\r\n", ["accept"] = "text/event-stream", ["cache-control"] = "no-cache", ["connection"] = "keep-alive" }, create = function(...) print(...) response = ... end }) print(master) print(err) print('yofff') end, url) copas.loop() local stored = {} local function enterFrame() local input,output = socket.select({ master }, nil, 0) if(response ~= stored) then stored = response print(stored) end if(input and #input \> 0) then print(input) end if(output and #output \> 0) then print(output) end end Runtime:addEventListener("enterFrame", enterFrame)

the latest iteration of my attempts.  still to no avail.  i’m also not sure if someone could move it to the network forum. i’m still pretty lost in trying to at least capture the partial table while it is listening for the event stream.

[rob]

The network.download()/.request() API’s should give you progress in your callback function. But generally the amount being downloaded needs to be known. The server has to convey this information to you before we can provide it.

Depending on the size of the dataset you’re downloading and if Firebase outputs JSON or not, you could just use network.request() and convert JSON results to a Lua table. No need to have a temporary file. Also if the temporary file is JSON then you can just JSON decode the file to get a Lua table.

Rob

[noriega]

The problem with the network.request/download, is that with the text/event-stream, the connection needs to be kept alive till it is cancelled and/or closed. The dataset is not a static number and is essentially always downloading/listening.  From my research, it’s like a websocket but it’s a one way connection after the initial call.  

network.download I figured uses a socket where it uses ltn12.sink.file to store the temp file as it downloads, then renames it to the appropriate filename defined in the params of params.response.filename   

i wanted to basically build the network.download but instead of saving the file to the ltn12.file, I wanted to save it to a ltn12.table 

[noriega]

local function getListener(event) if ( event.isError ) then print( "Network error: ", event.response ) else --outputs users.json file print (event.response) end end local function streamListener(event) if ( event.isError ) then print( "Network error: ", event.response ) else --workaround to see latest users.json file (called on each keep-alive, put, patch, auth\_revoked, and cancel) since we don't know the the event type                 network.request("https://project-xxxxx.firebaseio.com/users.json", "GET", getListener ) print (event.response) --is nil since not finished "downloading" end end local headers = {} headers["Accept"] = "text/event-stream" headers["Cache-Control"] = "no-cache" headers["Connection"] = "keep-alive" local params = {} params.headers = headers params.progress = "download" params.timeout = -1 params.handleRedirects = true network.request("https://project-xxxxx.firebaseio.com/users.json?sse=true", "GET", streamListener, params )

so i was able to receive events from firebase using a combination of the headers and putting a parameter called ?sse=true but I have nothing to reference from what triggered it.  (it can be keep alive, put, patch, auth_revoked, or cancel)

How hard would it be for the engineers to just to add the ability to see the the partial response within the “progress” phase?

event.response is nil until the get request is complete, but it will never be complete unless the stream is closed. (which we do not want)

this is an output of printing the (event) table on the streamListener function

table: 0x7fb02051b6f0 { utilities(285): [responseHeaders] =\> table: 0x7fb02051b6f0 { utilities(289): [Access-Control-Allow-Origin] =\> "\*" utilities(289): [Content-Type] =\> "text/event-stream" utilities(289): [Cache-Control] =\> "no-cache" utilities(287): } utilities(289): [phase] =\> "progress" utilities(291): [bytesEstimated] =\> -1 utilities(289): [name] =\> "networkRequest" utilities(291): [bytesTransferred] =\> 247 utilities(291): [status] =\> 200 utilities(289): [url] =\> "https://project-xxxx.firebaseio.com/users.json?sse=true" utilities(291): [isError] =\> false utilities(291): [requestId] =\> userdata: 0x7fb02051b788 utilities(302): }

https://firebase.google.com/docs/reference/rest/database/#section-streaming

event: event name data: JSON encoded data payload