network.request() as open source?

Solar2D Native builds
#1

I have a need of a HTTP client library which supports SOCKSv5.

Corona’s network module is such as a nice HTTP client library, i.e. fast and full of features, but there is no SOCKSv5 support.

To fix this I wrote my own non-blocking network module using luasocket and settimeout(0) but it is darn slow compared with Corona’s network module.

My question: Is Corona’s network module implemented as a plugin? If it were to be available as Open Source I could offer my time to add SOCKSv5 support. I could even add HTTP Keep-Alive support.

Any chance of that? Probably not.

I just have to write my own plugin duplicating the builtin ditto I suppose. :expressionless:

Cheers

/Joakim

#2

Perhaps I can propose an alternative solution?

You can make your socks5 plugin a local proxy server instead.  That way you can use our existing Lua network and socket APIs as-is, giving them a localhost URL/IP address and port number to your local proxy.  Another advantage of doing it this way is that your socks5 proxy server doesn’t have to be async anymore.  You can change it to be blocking since our Lua network request/response handling is already done on another thread.

#3

Hmm,

Do you mean that I can use the network module like this:

network.request("http://127.0.0.1:8080/http://www.dn.se/index.html", “GET”, networkListener)

On 127.0.0.1:8080 I then have an HTTP proxy server running. It takes the abs-path (http://www.dn.se/index.html) %C2%A0)of the URL and proxies traffic to it?

I could setup a HTTP proxy server such, as https://www.irif.fr/~jch/software/polipo/, as an Android service listening on 127.0.0.1:8080.

You propose that I “can make your socks5 plugin a local proxy server instead”. It could even be blocking because the network module uses threads. I lost you there. :slight_smile: Can I write a small blocking HTTP proxy as a C plugin(?) or a Lua plugin(?) and get it to run in the same thread context as each network.request(). You see. I am very confused. :slight_smile:

Thanks

/Joakim

#4

By proxy server, what I mean is that you would open up a local TCP socket connection which requests the operating system to give you an available TCP port number.  Once opened, you would use this localhost and port number via your network request API like this…

   http://127.0.0.1:<PortNumber>/<TargetUrl>

…or…

   https://127.0.0.1:<PortNumber>/<TargetUrl>

When the Corona app’s network request API opens a socket connection with your TCP socket server, your server would then open a SOCKv5 connection to <TargetUrl>.  From there, your proxy would act as a pass-through.  Meaning that the request sent from the Corona app’s network request to your TCP socket server would be passed through to the SOCKv5 connection you’ve opened up… and you would also do the reverse for the response by passing the SOCKv5 response back to the Corona app’s TCP socket connection.  A proxy server in this case is really just a software concept which you implement yourself.  That is, it’s a bridge.  You pass the received requests/responses between the SOCKv4 and SOCKv5 connections.  And this might be your only option for platforms whose native HTTP APIs don’t offer SOCKv5 support (like Win32).

On Android, you would want to listen in for local HTTP/HTTPS requests via the Java HttpUrlConnection class.  Unfortunately, it doesn’t give you raw access to the received HTTP request, meaning that you would have to reassemble the HTTP request the best you can when passing it through to the SOCKv5 connection.  The below link shows a Java code example that should give you the general idea on how to code it, but just note that’s it’s not a *great* example and you shouldn’t just copy-and-paste it.  :slight_smile:

   http://www.jtmelton.com/2007/11/27/a-simple-multi-threaded-java-http-proxy-server

Does this help?

#5

Thanks,

I understand. This is something I can do. It seems that https://www.irif.fr/~jch/software/polipo/ already has HTTP PROXY <-> SOCKSv5 bridging in place. It is another matter to dig into how to actually startup a background/service process on Android (and iOS).

I will stick with my non blocking luasocket “network” replacement until it hurts to much and then go for the solution above.

Thanks!

/Joakim

#6

giphy.gif

#7

Happy to help!

And also, you don’t have to set up your proxy server as a separate app service/process.  It’s merely a TCP socket you open up on a separate thread within your Corona app.

Case-in-point:

Corona’s native.newVideo() and media.playVideo() APIs on Android uses a local proxy server that we’ve coded in Java (within the same app process, but a different thread) for streaming Internet video.  The reason we do this is because the Android OS’ native VideoView Java class requires the web server hosting the video to support HTTP *partial* range requests (it’s used to seek to a certain times within the video stream).  In order for Corona to support video streaming from servers that don’t support HTTP partial range requests, our proxy server intercepts the Android OS’ HTTP request, changes the HTTP request by adding an “end” to the byte range (making it no longer a *partial* range request), and then passes the HTTP request through to the web server.

#8

Does this mean that I could write a C plugin which starts a pthread (or several) that implements a HTTP proxy, i.e. it/they calls bind() and listen() and then blocks on accept() waiting for an incoming http proxy request?

I am a bit shady when it comes to what kind of restrictions there exists when implementing C plugins. Is it OK to use pthreads? If so, is it OK to call poll()/select() in these pthreads?

In the C plugins I have written so far I have only done interfacing to third party libraries, i.e. libsodium and scrypt. No pthreads or networking needed.

In the best of worlds I would prefer to use luaproc (https://marketplace.coronalabs.com/plugin/luaproc) and then use blocking luasocket calls in each luaproc thread. This seems to be too easy. Would it be possible?

Thanks for your input on this!

Cheers

/Joakim

#9

It was too easy. :slight_smile:

https://forums.coronalabs.com/topic/66756-luaproc-can-not-load-socket-module-from-process/

/Joakim

#10

Sure, your plugin can use threads if it needs to.  There are no restrections there.  For example, our zip plugin use the C pthreads API to zip/unzip files.  Our network API uses Java threads on Android.  Just make sure that you don’t call Lua or UI APIs from another thread or else you’ll cause a race condition.  :slight_smile:

Corona Enterprise includes a “SimpleLuaExternsion” sample project which shows you how to create an async Lua API that calls a Lua listener once the threaded operation has finished.  Check out its “AsyncCallLuaFunction.java” source file to see how to do this on Android.

The luaproc plugin is 3rd party.  So, unfortunately, I don’t know what is and isn’t possible with it.

#11

Perhaps I can propose an alternative solution?

You can make your socks5 plugin a local proxy server instead.  That way you can use our existing Lua network and socket APIs as-is, giving them a localhost URL/IP address and port number to your local proxy.  Another advantage of doing it this way is that your socks5 proxy server doesn’t have to be async anymore.  You can change it to be blocking since our Lua network request/response handling is already done on another thread.

#12

Hmm,

Do you mean that I can use the network module like this:

network.request("http://127.0.0.1:8080/http://www.dn.se/index.html", “GET”, networkListener)

On 127.0.0.1:8080 I then have an HTTP proxy server running. It takes the abs-path (http://www.dn.se/index.html) %C2%A0)of the URL and proxies traffic to it?

I could setup a HTTP proxy server such, as https://www.irif.fr/~jch/software/polipo/, as an Android service listening on 127.0.0.1:8080.

You propose that I “can make your socks5 plugin a local proxy server instead”. It could even be blocking because the network module uses threads. I lost you there. :slight_smile: Can I write a small blocking HTTP proxy as a C plugin(?) or a Lua plugin(?) and get it to run in the same thread context as each network.request(). You see. I am very confused. :slight_smile:

Thanks

/Joakim

#13

By proxy server, what I mean is that you would open up a local TCP socket connection which requests the operating system to give you an available TCP port number.  Once opened, you would use this localhost and port number via your network request API like this…

   http://127.0.0.1:<PortNumber>/<TargetUrl>

…or…

   https://127.0.0.1:<PortNumber>/<TargetUrl>

When the Corona app’s network request API opens a socket connection with your TCP socket server, your server would then open a SOCKv5 connection to <TargetUrl>.  From there, your proxy would act as a pass-through.  Meaning that the request sent from the Corona app’s network request to your TCP socket server would be passed through to the SOCKv5 connection you’ve opened up… and you would also do the reverse for the response by passing the SOCKv5 response back to the Corona app’s TCP socket connection.  A proxy server in this case is really just a software concept which you implement yourself.  That is, it’s a bridge.  You pass the received requests/responses between the SOCKv4 and SOCKv5 connections.  And this might be your only option for platforms whose native HTTP APIs don’t offer SOCKv5 support (like Win32).

On Android, you would want to listen in for local HTTP/HTTPS requests via the Java HttpUrlConnection class.  Unfortunately, it doesn’t give you raw access to the received HTTP request, meaning that you would have to reassemble the HTTP request the best you can when passing it through to the SOCKv5 connection.  The below link shows a Java code example that should give you the general idea on how to code it, but just note that’s it’s not a *great* example and you shouldn’t just copy-and-paste it.  :slight_smile:

   http://www.jtmelton.com/2007/11/27/a-simple-multi-threaded-java-http-proxy-server

Does this help?

#14

Thanks,

I understand. This is something I can do. It seems that https://www.irif.fr/~jch/software/polipo/ already has HTTP PROXY <-> SOCKSv5 bridging in place. It is another matter to dig into how to actually startup a background/service process on Android (and iOS).

I will stick with my non blocking luasocket “network” replacement until it hurts to much and then go for the solution above.

Thanks!

/Joakim

#15

giphy.gif

#16

Happy to help!

And also, you don’t have to set up your proxy server as a separate app service/process.  It’s merely a TCP socket you open up on a separate thread within your Corona app.

Case-in-point:

Corona’s native.newVideo() and media.playVideo() APIs on Android uses a local proxy server that we’ve coded in Java (within the same app process, but a different thread) for streaming Internet video.  The reason we do this is because the Android OS’ native VideoView Java class requires the web server hosting the video to support HTTP *partial* range requests (it’s used to seek to a certain times within the video stream).  In order for Corona to support video streaming from servers that don’t support HTTP partial range requests, our proxy server intercepts the Android OS’ HTTP request, changes the HTTP request by adding an “end” to the byte range (making it no longer a *partial* range request), and then passes the HTTP request through to the web server.

#17

Does this mean that I could write a C plugin which starts a pthread (or several) that implements a HTTP proxy, i.e. it/they calls bind() and listen() and then blocks on accept() waiting for an incoming http proxy request?

I am a bit shady when it comes to what kind of restrictions there exists when implementing C plugins. Is it OK to use pthreads? If so, is it OK to call poll()/select() in these pthreads?

In the C plugins I have written so far I have only done interfacing to third party libraries, i.e. libsodium and scrypt. No pthreads or networking needed.

In the best of worlds I would prefer to use luaproc (https://marketplace.coronalabs.com/plugin/luaproc) and then use blocking luasocket calls in each luaproc thread. This seems to be too easy. Would it be possible?

Thanks for your input on this!

Cheers

/Joakim

#18

It was too easy. :slight_smile:

https://forums.coronalabs.com/topic/66756-luaproc-can-not-load-socket-module-from-process/

/Joakim

#19

Sure, your plugin can use threads if it needs to.  There are no restrections there.  For example, our zip plugin use the C pthreads API to zip/unzip files.  Our network API uses Java threads on Android.  Just make sure that you don’t call Lua or UI APIs from another thread or else you’ll cause a race condition.  :slight_smile:

Corona Enterprise includes a “SimpleLuaExternsion” sample project which shows you how to create an async Lua API that calls a Lua listener once the threaded operation has finished.  Check out its “AsyncCallLuaFunction.java” source file to see how to do this on Android.

The luaproc plugin is 3rd party.  So, unfortunately, I don’t know what is and isn’t possible with it.