This is a continuation of the discussion from the API page.
http://developer.anscamobile.com/reference/index/networkdownload [import]uid: 7559 topic_id: 24298 reply_id: 324298[/import]
Alright everybody,
we have had a lot of negative feedback that our downloads are not working. We download additional content (pngs/mp3s) via network.download.
After the download is complete we calculate a sha1 over the file to ensure that the dl went ok. (OpenAL is not very forgiving with corrupt files ;))
What we noticed is, that sometimes the hashes are wrong (reproducible wrong) and we couldn’t figure out why. But on using WIFI everything works perfectly fine. Strange ey?
After running some test I found out that the mobile-network-downloaded file has gotten compressed. (From 21kb down to 5kb.) and this is why the hash fails.
Yes, so, I had the bug reproduced on a GT-I9100 T-Mobile USA 2.3.3 with EDGE and only with EDGE.
To me it seems like some providers use a transparent proxy on port 80 to “enhance” the browsing experience of their users by compressing web content.
So do a hash if you want to be sure to have HQ images and not some compressed junk.
[import]uid: 11772 topic_id: 24298 reply_id: 98206[/import]
Thanks for the warning!
How do you do the hash on the png loaded?
Something like this?
-
Generate the sha1 for each png you will later download ()
Use a service like http://hash.online-convert.com/sha1-generator for this? -
Store these sha1 values in your code
-
Download a png using network.download() and store it in the system.DocumentsDirectory
-
Read the png into a string using file io like this:
[lua]local path = system.pathForFile( “my.png”, system.DocumentsDirectory )
local fh, reason = io.open( path, “r” )
if fh then
local contents = fh:read( “*a” )
end
io.close( fh )[/lua]
- find the hash for the loaded string like this
[lua]local hash = crypto.digest( crypto.sha1, contents )[/lua]
- compare the generated hash with the hash already stored in the code for “my.png”
- if hash is fine, load & display “my.png”
My questions:
* is this the right way to do it? It seems that for this I have to load the png 2 times, first for reading it into a string to compare it with the hash, then again to display it. This can take some time using big pngs…
* hm, maybe put the name of the file to download and the corresponding hash value into an XML file, so the hash values for all the different pngs must not be stored in the code from the beginning (if you want to provide new content)
I have to code something like this in the next few days, so a hint would be appreciated.
Thanks,
Andreas
[import]uid: 107675 topic_id: 24298 reply_id: 98221[/import]
Instead of using network.download it might be a good idea to create a custom function for downloading using the FTP protocol (with Lua sockets). Might bypass the transparent compression problem, but it will require using coroutines if you don’t want to block the application while the files are downloading. [import]uid: 61899 topic_id: 24298 reply_id: 98224[/import]
this is exactly how I did it. You are right, the first file of the download could be an xml file with all the other files and hashes. I did not do that because if I change the content I might have to change the code as well and it’s just a lot of stuff which could go wrong again - so my hashes are hardcoded in a table.
Actually it’s a hashmap with the file, version and hash like:
[lua]…
TheAllKnowingTable[“Myfile.png”] = {}
TheAllKnowingTable[“Myfile.png”].hash = “212a23aeea23723…”)
TheAllKnowingTable[“Myfile.png”].ver = 1.2
…other stuff[/lua]
And I use recursion to iterate over it to get the network.download synchronous to download them one by one and don’t open 100 connections.
This far it is working perfectly fine and the performance is okay too.
Some thoughts:
*use pcall when using crypto.digest and check if contents is not nil.
*if the web server gives you a 404, 401 or whatever the “my.png” will become an xml file (depends on the server though) stating that error. Handle this error. Parse the file for those well known codes if the hash fails. It will save you a lot of searching. The event.isError in the callback will not be set if something like that happens.
*but also handle the event.isError from the network listener properly. One which is very likely is: No space left on device. (Or timeouts as well) communicate everything to the user or retry the download a number of times. Whatever makes sense. If Ansca would have documented this better it would be easier.
@CluelessIdeas
yes FTP would another possibility but … I guess it will cause more trouble than it solves because it’s such a pain in the neck… and I don’t know if providers block the ftp port. I would say that FTP is really the last resort, I would use network.download and try changing the ports from 80 to something else if the hashes go wrong and see where I can get trough. If I can’t, just tell the user to go on WIFI because the provider “filters” the traffic.
[import]uid: 11772 topic_id: 24298 reply_id: 98247[/import]
Thank you for your thoughts - there really are some traps.
I’m positive now that I can come up with a solution like this in reasonable time, but I first will try another solution - using a native WebPopup in my app for showing infos about other apps I did and linking to these apps (instead of loading all the app icons and the corresponding xml via network download).
It will be painless in comparison, I guess.
Best,
Andreas
[import]uid: 107675 topic_id: 24298 reply_id: 98284[/import]
Dear All,
Please help me…
Im using network.download to get images but always download some empty or crashed (only 4k) images, randomly about 3-4 pcs from 20 images…
local headers = {}
headers[“Content-Type”] = “application/jpeg”
headers[“Accept-Language”] = “en-US”
local postData = “loginid=” … mime.b64(“user”) … “&password=” … mime.b64(“password”) … “&fname=” … mime.b64(fname) … “&mode=” … mime.b64(“pics”)
local params = {}
params.body = postData
network.download(
“http://remotehost/download.php”,
“POST”,
networkListener,
params,
fname,
system.DocumentsDirectory )
Do you have any idea?
Thank you in advance,
Zsolt [import]uid: 126421 topic_id: 24298 reply_id: 98831[/import]
Hi aleeri,
Does the proxy compress when using https instead of clean http?
On the other hand, if I remember correctly in Python the httplib supports compression, but it handles compression behind the scenes, so your code reiceve the file uncompressed, so maybe it’s a bug in Corona. Did you open a bug request?
I need to download large images for my app, so I really hope you can solve this issue and share the solution with the community.
Best regards
[import]uid: 99513 topic_id: 24298 reply_id: 105835[/import]
No, a proxy can not compress https traffic since it is encrypted. But you will need to have a valid ssl certificate, or the phone will not accept https traffic. (It will not accept a self-signed certificate as well.)
It is definitely not a bug in Corona since the http request does not know what the file will look like and a transparent proxy can not be detected. This is why it is important to do hashes. (btw. do a sha1)
The solution which I came up with is the following:
- download data on port 80
- hash the file, if the hash fails, store hash
- download again on port 80
- hash the file, if the hash fails again - and this hash == old hash -> proxy detected
- change port to 1234 (or something else)
- start over till a non proxy port is found
If you can’t find an open/non proxied port tell the user what the problem is and to use WIFI.
I am doing quite well with this workaround since most operators only proxy port 80. If you want to be 100% save, buy the cert and do ssl.
Best,
Alex [import]uid: 11772 topic_id: 24298 reply_id: 105854[/import]
Hi Alex,
Thank you very much for your prompt reply.
I’m still thinking that there could be a problem with Corona http lib. The proxy (transparent or not) must add “Content-Encoding: gzip” to the response or the web browser wouldn’t know it has to decompress the content.
Anyway, thank you again. I will test with https.
Best regards
JF [import]uid: 99513 topic_id: 24298 reply_id: 105880[/import]
How I can disable the cache mechanism of network.download?
Because on mac os and IOS, when I change my http server content (e.g. big images), the files I get by network.download remains the same as before.
And this bug also happens in showWebPopup, when I changed my webpage back group image, the image opened by showWebPopup remains the same as before.
[import]uid: 35642 topic_id: 24298 reply_id: 107834[/import]
network.download, downloads into a baseDir.
With the new Lua File System. Is it possible to get network.download to download Directly into a SUB directory that we have created with LFS?
i.e. I have created a sub directory in system.DocumentsDirectory and i want to download graphics directly into the sub directory i created. right now when i try and feed it a path other than system base directories, it ignors them and reverts to the default of system.DocumentsDirectory.
[import]uid: 88147 topic_id: 24298 reply_id: 110454[/import]
You should be able to download into a subdirectory. The baseDir needs to stay system.DocumentesDirectory but you add the subdirectory name to the destFile parameter. [import]uid: 7559 topic_id: 24298 reply_id: 110509[/import]
“How I can disable the cache mechanism of network.download?
Because on mac os and IOS, when I change my http server content (e.g. big images), the files I get by network.download remains the same as before.”
Same problem… what can we do to force devices (and mac emulator) to not use cached file and to download the new version ?
[import]uid: 107268 topic_id: 24298 reply_id: 115675[/import]
How to check whether the image has been downloaded when using network.download()?
[code]
local linkForImages = {
“http://i1164.photobucket.com/albums/q576/rahulsingh2k10/Story1/”,
“http://i1164.photobucket.com/albums/q576/rahulsingh2k10/Story2/”,
“http://i1164.photobucket.com/albums/q576/rahulsingh2k10/Story3/”,
“http://i1164.photobucket.com/albums/q576/rahulsingh2k10/Story4/”
}
local namesForImageinEachStory =
{
{“menu1.jpg”, “game1.png”, “record1.png”, “read1.png”, “home1.png”},
{“menu2.jpg”, “game2.png”, “record2.png”, “read2.png”, “home2.png”},
{“menu3.jpg”, “game3.png”, “record3.png”, “read3.png”, “home3.png”},
{“menu4.jpg”, “game4.png”, “record4.png”, “read4.png”, “home4.png”}
}
for i = 1 ,5 do
local path = system.pathForFile( namesForImageinEachStory[tableIndex][i], system.DocumentsDirectory )
local myFile = io.open( path, “w+b” )
network.download( linkForImages[tableIndex]…namesForImageinEachStory[tableIndex][i], “GET”, showImage, namesForImageinEachStory[tableIndex][i], system.DocumentsDirectory )
end
[/code] [import]uid: 45566 topic_id: 24298 reply_id: 135572[/import]
How to check whether the image has been downloaded when using network.download()?
[code]
local linkForImages = {
“http://i1164.photobucket.com/albums/q576/rahulsingh2k10/Story1/”,
“http://i1164.photobucket.com/albums/q576/rahulsingh2k10/Story2/”,
“http://i1164.photobucket.com/albums/q576/rahulsingh2k10/Story3/”,
“http://i1164.photobucket.com/albums/q576/rahulsingh2k10/Story4/”
}
local namesForImageinEachStory =
{
{“menu1.jpg”, “game1.png”, “record1.png”, “read1.png”, “home1.png”},
{“menu2.jpg”, “game2.png”, “record2.png”, “read2.png”, “home2.png”},
{“menu3.jpg”, “game3.png”, “record3.png”, “read3.png”, “home3.png”},
{“menu4.jpg”, “game4.png”, “record4.png”, “read4.png”, “home4.png”}
}
for i = 1 ,5 do
local path = system.pathForFile( namesForImageinEachStory[tableIndex][i], system.DocumentsDirectory )
local myFile = io.open( path, “w+b” )
network.download( linkForImages[tableIndex]…namesForImageinEachStory[tableIndex][i], “GET”, showImage, namesForImageinEachStory[tableIndex][i], system.DocumentsDirectory )
end
[/code] [import]uid: 45566 topic_id: 24298 reply_id: 135572[/import]
I’m in the same case as team1:
“I have created a sub directory in system.DocumentsDirectory and i want to download graphics directly into the sub directory i created. right now when i try and feed it a path other than system base directories, it ignors them and reverts to the default of system.DocumentsDirectory”
As Tom said, I’ve tryed to add the subdirectory name to the destFile parameter, but the files are saved out of this subfolder.
What am I doing wrong?
Alberto. [import]uid: 44013 topic_id: 24298 reply_id: 143201[/import]
I’m in the same case as team1:
“I have created a sub directory in system.DocumentsDirectory and i want to download graphics directly into the sub directory i created. right now when i try and feed it a path other than system base directories, it ignors them and reverts to the default of system.DocumentsDirectory”
As Tom said, I’ve tryed to add the subdirectory name to the destFile parameter, but the files are saved out of this subfolder.
What am I doing wrong?
Alberto. [import]uid: 44013 topic_id: 24298 reply_id: 143201[/import]
This is how I try:
local docs\_path = system.pathForFile( "", system.DocumentsDirectory )
-- change current working directory
local success = lfs.chdir( docs\_path )
local new\_folder\_path
local dname = "proba"
if success then
lfs.mkdir( dname )
new\_folder\_path = lfs.currentdir() .. "/" .. dname .. "/"
end
...
local destino = system.pathForFile("",system.DocumentsDirectory)
lfs.chdir(destino)
local dest = destino.."/proba/"
network.download( origin, "GET", arqDescargado, arquivo, dest )
...
The files are downloaded to DocumentsDirectory instead of DocumentsDirectory/proba
What am I doing wrong? [import]uid: 44013 topic_id: 24298 reply_id: 144157[/import]
The problem with your code example is “dest” is defined as “baseDir”, which is a file directory constant and should not be appended with your subfolder name. The subfolder should be appended to the file name instead
[lua]network.download( origin, “GET”, arqDescargado, “/proba/” … arquivo, system.DocumentsDirectory )[/lua] [import]uid: 7559 topic_id: 24298 reply_id: 144188[/import]