Android file download

antkmk · June 3, 2022, 4:32pm

I’ve been beating my head against this one for awhile and hope someone can help. Here’s the facts:

(1) I have an app that uses “network.download”.
(2) The code works in the simulator and on my iPhone
(3) The code does NOT work on Android
(4) I can load the URL in a browser window on my Mac and using a browser on my Android phone
(5) The URL is: [https://]graphical.weather.gov/xml/sample_products/browser_interface/ndfdBrowserClientByDay.php?whichClient=NDFDgenByDayLatLonList&listLatLon=44.112733%2C-73.923725&format=24+hourly&startDate=2022-06-03&numDays=4&Unit=e&Submit=Submit
(6) The file that returns from this is an XML file that I store locally in the “system.DocumentsDirectory”
(7) The file is returned, stored, read, and parsed correctly in the simulator and on my iPhone.

(8) On Android I get back an HTML formatted response that says the following:
[H1]Access Denied[/H1]
You don’t have permission to access “http://graphical.weather.gov/xml/sample_products/browser_interface/ndfdBrowserClientByDay.php?” on this server.
Reference #18.7fee2117.1654272202.49a15e42"

(9) I added an “xml” folder to my “AndroidResources / res” directory with a new file - “network_security_config.xml” with the following (not because I know what I’m doing with this file, but because I came across something similar in my search for a solution):
[?xml version=“1.0” encoding=“utf-8”?]
[network-security-config]
[domain-config cleartextTrafficPermitted="true]>
[domain includeSubdomains=“true”]https://graphical.weather.gov/[/domain]
[domain includeSubdomains=“true”]http://graphical.weather.gov/[/domain]
[domain includeSubdomains=“true”]graphical.weather.gov[/domain]
[domain includeSubdomains=“true”]graphical.weather.gov/xml/sample_products/browser_interface/
[/domain-config]
[/network-security-config]

(10) The version of Solar2d that I’m using is: 2022.3673

Help! Please! And thank you in advance. ~Keith

P.S. Note: I’m using square brackets to prevent HTML formatting. In the code above they replace angled brackets.

roaminggamer · June 3, 2022, 5:53pm

Please share the contents of your build.settings file here.
You can use three back-ticks (same key as ~ on keyboard) surrounding code to avoid that formatting issue you had above.

Screenshot in post-editor:

Output looks like this:

This is multi-line code 
with three back-ticks before
and after

There was a recent discussion like this in the ‘help-and-support’ thread on Discord.
See discussion starting May 31 2022 by iirzd and continued with Xedur

Discussion starts like this and continues to more info:

Have you requested permission to use the Internet on Android, in build.settings?
[1:59 AM]
On a different note, is anyone familiar with using Solar2D apps to open/launch other Solar2D apps on iOS/Android?

Perhaps via Android intents and/or:

Solar2D Documentation — API Reference | Libraries | system | canOpe…
Solar2D lets you build games/apps for all major platforms including iOS, Android, Kindle, Apple TV, Android TV, macOS, and Windows. Get the free toolset!

@XeduR
Have you requested permission to use the Internet on Android, in build.settings?

iirzd — 06/01/2022
Yes, however when I checked which permissions the app had in settings, the permission was there but not set as being used. I only saw the network permission when I viewed “All permissions” which the app is capable of.

@iirzd
Yes, however when I checked which permissions the app had in settings, the permission was there but not set as being used. I only saw the network permission when I viewed “All permissions” which the app is capable of.

antkmk · June 3, 2022, 6:43pm

	-- 
	-- Android section
	--	
	android =
   {
      versionCode = "260", -- app's actual version (release use 1 = 1.0; 12 = 1.2 - decimals not allowed)
      versionName = "2.6.0", -- shown to user
      usesPermissions =
         {
			"android.permission.CAMERA",
         "android.permission.INTERNET",
         "android.permission.WRITE_EXTERNAL_STORAGE",
         "android.permission.READ_EXTERNAL_STORAGE",
         "android.permission.ACCESS_FINE_LOCATION",
         "android.permission.ACCESS_COURSE_LOCATION",
         },
		usesFeatures =
		   {
			{ name = "android.hardware.camera", required = false },
			{ name = "android.hardware.camera.front", required = false },
			{ name = "android.hardware.location", required = true },
         { name = "android.hardware.location.gps", required = true },
         { name = "android.hardware.location.network", required = true },
         { name = "android.hardware.sensor.compass", required = true },
		   },
   },

antkmk · June 3, 2022, 7:11pm

Thank you…

That thread, which leads back to the Solar2d forum is where I found the “network_security_config.xml” settings that I tried.

And my build settings which I just listed are also set. Still Android is not happy!

P.S. Thank you for the info on commenting code!

roaminggamer · June 3, 2022, 7:19pm

OK. I don’t see anything specifically wrong with that.

I’d like to see more details about the error. I suggest trying this to get more details.

1.Download this recent post/answer: https://github.com/roaminggamer/RG_FreeStuff/raw/master/AskEd/2022/05/fileIO_userDB_example.zip

Take the scripts/extensions folder and copy it to ~/extensions in your project.
Require the extensions listed below near the top of main.lua (before running your app code)

-- Require these just once in main.lua
require "extensions.string"
require "extensions.io"
require "extensions.table"

Write a listener like this and pass it to your network.download call.

local function networkListener( event )
    -- This one function call will do an exhaustive parse and dump of the event to your console/log
    table.print_r(event)
end

See here for more details on listeners etc.

See if the dump gives any more details on why the request was denied and if so share them with us.

Hopefully with your build.settings and the additional log data someone will see the issue.

antkmk · June 3, 2022, 7:31pm

Here’s the output:

Jun 03 15:28:47.978 moto g(7): table: 0x7ca2cdffc0 
Jun 03 15:28:47.979 {
Jun 03 15:28:47.980 moto g(7):  [responseHeaders] => table: 0x7ca2cdffc0
                    moto g(7):  {
                    moto g(7):     [Connection] => "close"
                    moto g(7):     [Content-Length] => "364"
                    moto g(7):     [X-Android-Response-Source] => "NETWORK 403"
                    moto g(7):     [Date] => "Fri, 03 Jun 2022 19:28:47 GMT"
                    moto g(7):     [X-Android-Sent-Millis] => "1654284532650"
                    moto g(7):     [Expires] => "Fri, 03 Jun 2022 19:28:47 GMT"
Jun 03 15:28:47.990 moto g(7):     [X-Android-Selected-Protocol] => "http/1.1"
                    moto g(7):     [Strict-Transport-Security] => "max-age=31536000 ; includeSubDomains ; preload"
                    moto g(7):     [X-Android-Received-Millis] => "1654284532690"
                    moto g(7):     [Content-Type] => "text/html"
                    moto g(7):     [Mime-Version] => "1.0"
                    moto g(7):     [HTTP-STATUS-LINE] => "HTTP/1.1 403 Forbidden"
                    moto g(7):     [Server] => "AkamaiGHost"
                    moto g(7):  }
                    moto g(7):  [responseType] => "text"
                    moto g(7):  [phase] => "ended"
Jun 03 15:28:47.991 moto g(7):  [bytesEstimated] => 364
                    moto g(7):  [response] => "<HTML><HEAD>
                    moto g(7): <TITLE>Access Denied</TITLE>
                    moto g(7): </HEAD><BODY>
                    moto g(7): <H1>Access Denied</H1>
                    moto g(7):  
                    moto g(7): You don't have permission to access 
"http&#58;&#47;&#47;graphical&#46;weather&#46;gov&#47;xml&#47;sample&#95;products&#47;browser&#95;interface&#47;ndfdBrowserClientByDay&#46;php&#63;" on this server.<P>
                    moto g(7): Reference&#32;&#35;18&#46;7fee2117&#46;1654284527&#46;4b29c139
                    moto g(7): </BODY>
                    moto g(7): </HTML>
                    moto g(7): "
                    moto g(7):  [name] => "networkRequest"
                    moto g(7):  [bytesTransferred] => 364
                    moto g(7):  [status] => 403
                    moto g(7):  [url] => "https://graphical.weather.gov/xml/sample_products/browser_interface/ndfdBrowserClientByDay.php?whichClient=NDFDgenByDayLatLonList&listLatLon=44.112733%2C-73.923725&format=24+hourly&startDate=2022-06-03&numDays=4&Unit=e&Submit=Submit"
                    moto g(7):  [isError] => false
                    moto g(7):  [requestId] => false
                    moto g(7): }

roaminggamer · June 3, 2022, 8:19pm

I’ve got to mull this over for a bit and in the meantime I hope others also look at this and make suggestions.

The fact that you can access the URL from a browser on the same device tells me it’s likely something in the request that the server doesn’t like. What that is puzzles me right now.

antkmk · June 3, 2022, 9:00pm

Thank you.
I saw “ AkamaiGHost” and thought maybe it was a certificate thing? But building and installing on the device with and without the solar2d debug certificate made no difference. So, I’m at a loss. I appreciate your help with this. I’ve added the details of this to a recent thread that Vlad was on also, so hopefully he may also respond.

antkmk · June 5, 2022, 12:45pm

A little more information that may help:
The URL: I’m calling is:

https://graphical.weather.gov/xml/SOAP_server/ndfdSOAPclientByDay.php?whichClient=NDFDgenByDay&lat=44.11&lon=-73.923725&format=24+hourly&startDate=2022-06-04&numDays=3&Unit=e&Submit=Submit

I also replaced the URL in the sample app code in “AsyncHTTP” with the URL above and I get the same thing… a 403 access denied.

This URL works in the simulator. It works on an iOS device. And it works if I enter it in a browser on my Android device. But it doesn’t work in an app on an Android device.

~Keith

antkmk · June 7, 2022, 12:11pm

Hello @roaminggamer, or anyone else who can help!
I’ve still had no luck with the network.download command on Android. It seems the server doesn’t like my headers - at least that’s the current theory, but only when called from the app built for Android. My headers and the URL are below.
What I’m wondering is is there any way to see what the server thinks my headers are when the call arrives, or in other words, is it possible that Solar2d is changing or ignoring my header settings when the app is built for Android? Is there any way I can tell if this is the case?

"https://graphical.weather.gov/xml/sample_products/browser_interface/ndfdBrowserClientByDay.php?listLatLon=44.11,-73.923725&format=24+hourly&numDays=3"

Here’s the headers I’m using:

   headers["Accept"]          = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
   headers["Host"]          = "graphical.weather.gov"
   headers["User-Agent"]      = "ADK46erNow/v2.6.0 (http://onebadant.com; ant@onebadant.com)"
   headers["Accept-Language"] = "en-US,en;q=0.9"
   headers["Accept-Encoding"] = "gzip, deflate, br"
   headers["Connection"] = "keep-alive"

Thank you in advance. ~Keith

P.S. This just in… I’ve been on the gov weather forum and they were able to see that my user-agent header was being set back to the default Android user agent. Meaning that it’s ignoring the user-agent I’m specifically setting. This sounds like a bug or something that needs to be fixed in Solar2d’s code, unless there’s a setting I can make that specifically tells the app to use my user-agent setting and not change or ignore it.

antkmk · June 7, 2022, 3:16pm

Anyone following this thread or who runs into this issue - @vlads solved this!
(Thank you again Vlads and also thank you @roaminggamer for your help with this!)
Here’s how to get the download from the weather.gov service to work:
(1) First forget all the headers above,
(2) Use this:

weatherFileName = 'weather.xml'
weatherparser = { weatherPathFileName = 'https://graphical.weather.gov/xml/sample_products/browser_interface/ndfdBrowserClientByDay.php?listLatLon=44.11,-73.923725&format=24+hourly&numDays=3'}


local params = { 
	headers = {
		["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.63 Safari/537.36",
	},
}
  
network.download( weatherparser.weatherPathFileName, "GET", XMLNetworkListener, params, weatherFileName, system.DocumentsDirectory )