Can't read Http Get response when is Content-Encoding: gzip

Solar2D Network
#1

Hi Solar2D Community! :wave:

I need to get a file using an http request. The 3rd party server response’s is a json text which contains theContent-Encoding: gzip and Content-Type: application/json headers.

I can’t correctly read the response’s json content as it’s misinterpreted by the http corona lib.

is this something that solar2d is not supporting?
to process a gzipped response with content-type application/json?

has anyone achieved that successfully before?

I 've found some very old related-ish topics here and there
(Invalid File Signature error - #17 by vitaly1)

but nothing specially clear about the support of this kind of responses.

thank you

#2

Hello and welcome!

If the data is compressed in a gzip (or any format or even encrypted) then you’ll need process said data to get its content, in this case the json data.

Solar2d currently doesn’t have a way to extract gzip, although we have a zip plugin, it extracts filenames and cannot just extract “data”.

I prefer plugins or pure Lua solutions, and it seems the latter is the only option at the moment…

You’ll basically need to download the gzip data to a temp directory and then use something like zzlib to decompress said data. If you can manage to get all of the gzip data in a string from your HTTP session then you can skip the part about downloading it to disk, and zzlib can still get the job done.

If you do end up using zzlib then you’ll just need zzlib.lua and inflate-bit32.lua. The latter needs to be modified to include the bit plugin from Solar2d:

-- change this on line 14
local bit = bit32 or bit

-- to this
local bit = require( "plugin.bit" )

You’ll also want to add it to the plugins sub-table in your project’s build.settings file:

plugins =
    {
        ["plugin.bit"] =
        {
            publisherId = "com.coronalabs"
        },
    },

Here’s working code I have tested from what I mentioned above:

local json = require("json")
local zzlib = require("zzlib")

local function networkListener( event )
	if ( event.isError ) then
		print( "Network error - download failed: ", event.response )
	elseif ( event.phase == "began" ) then
		print( "Progress Phase: began" )
	elseif ( event.phase == "ended" ) then
		print( "Processing data ..." )
		-- If parsing a string then use zzlib.gunzip, for a file on disk use zzlib.gunzipf
		local decompressedData = zzlib.gunzipf(system.pathForFile(event.response.filename, system.TemporaryDirectory ))
		print("JSON Data:", decompressedData)
		
		local luaTable = json.decode(decompressedData)
		print("Table Data:")
		for k,v in pairs(luaTable) do print(k, v) end
	end
end

local params = {progress = true}

network.download(
	"https://httpbin.org/gzip",
	"GET",
	networkListener,
	params,
	"data.gzip",
	system.TemporaryDirectory
)

EDIT:
Here’s working code for parsing data directly without saving to disk:

local json = require("json")
local zzlib = require("zzlib"
local socket = require("socket")
local http = require("socket.http")
local ltn12 = require("ltn12")

local url = "https://httpbin.org/gzip"

local response_body = {}  -- Table to store response chunks

local _, status_code, _, status_line = http.request{
  url = url,
  sink = ltn12.sink.table(response_body),
}

if status_code == 200 then
  local response_text = table.concat(response_body) -- Combine all chunks into a single string
  print("Status Code:", status_code)
  print("Response Body:", response_text)

  -- Process the response data as needed
  print( "Processing data ..." )
  -- If parsing a string then use zzlib.gunzip, for a file on disk use zzlib.gunzipf
  local decompressedData = zzlib.gunzip(response_text)
  print("JSON Data:", decompressedData)

  local luaTable = json.decode(decompressedData)
  print("Table Data:")
  for k,v in pairs(luaTable) do print(k, v) end
else
  print("Request failed:", status_code) -- You can handle errors here
end

Hope that helps!

#3

Hey Siu, thanks for your detailed response…

I am doing a

network.request( ... )

and I am passing params: { headers: { 'Accept-Encoding': 'gzip' } }
because if not the server would return a non gzipped version of the content.

The problem is the content is large enough so I must receive a gzipped version of the content.

when doing this… I end up having a binary response that’s treated as text
I was looking at the solar2d code… and I think that the reason lies in this piece of code

if ( ( NULL != contentEncoding ) || ( ( NULL != contentType ) && isContentTypeText( contentType ) ) )
			{
				// If the Content-Type has a charset, or if it is "texty", then let's treat it as text
				debug("treating content as text");
				body->bodyType = TYPE_STRING;
				body->bodyString = new UTF8String( );
				body->bodyString->reserve( contentLength > 0 ? contentLength : defaultContentAllocation );
			}
			else
			{
				debug("treating content as binary");
				body->bodyType = TYPE_BYTES;
				body->bodyBytes = new ByteVector( );
				body->bodyBytes->reserve(  contentLength > 0 ? contentLength : defaultContentAllocation  );
			}

content encoding is defined in the response (gzip) but because of that the solar2d code will treat the response as text.

once the binary is treated as utf8 then I am screwed :frowning_face:

I wish I weren’t forced to download that content before

do you think I could use the last piece of code you propose inside corona?
the one using ltn and socket.http?

local json = require("json")
local zzlib = require("zzlib"
local socket = require("socket")
local http = require("socket.http")
local ltn12 = require("ltn12")
#4

You might be correct on that piece of code as it seems to be formatting the data for you.

The second code snippet I provided should work as you would be using the Socket library instead of the Network library; the incoming data is raw data and it’s up to you to manipulate it as desired.

It’s worth a shot, and if you need further troubleshooting please let us know. :slight_smile: