Uploading files to Parse.com

Hi all,

Just wondering if anyone has had any success uploading files to Parse using their REST API? I have been battling with it and as of yet, am unable to get it working. Unfortunately their documentation is sparse and does not really say much about content encoding or how to do it ( https://www.parse.com/docs/rest#files ) . I have my code working for text files but am unable to get it working for images and sound files… I.E its an ecoding problem

If anyone has managed it or has any pointers it would be greatly appreciated! I have included my function that I have built on to “Parse for Corona SDK” by Dotnaught . I have tried changing the content type as well as base 64 encoding the file and many variations in between, but with no luck

Thanks

Antony

[lua]local function uploadFile (fileName)

local mime = require “mime”
local path = system.pathForFile( fileName )
print(path)
local fileHandle = io.open( path, “rb” )
if fileHandle then

headers[“Content-Type”] = “image/jpeg”

–params.body = mime.b64( fileHandle:read( “*a” ) )
params.body = fileHandle:read( “*a” )

io.close( fileHandle )
request = “uploadFile”
print(“Uploading File”)
network.request( baseUrl … class.files … “/” … fileName, “POST”, networkListener, params)
end
end[/lua] [import]uid: 169392 topic_id: 31044 reply_id: 331044[/import]

Hey Antony

I got it working some time ago. Here is the piece of code I used.
Good luck!

[lua] local json = require (“json”)
local mime = require “mime”

local parse = {}
local headers = {}

headers[“Content-Type”] = “application/json”
headers[“X-Parse-Application-Id”] = “???”
headers[“X-Parse-REST-API-Key”] = “???”

local params = {}
params.headers = headers


– Upload file

local upUrl = “https://api.parse.com/1/files/

function parse.upload ( fileName, callback )

params.headers[“Content-Type”] = “multipart/text”

local path = system.pathForFile( fileName, system.DocumentsDirectory )
local fileHandle = io.open( path, “rb” )
local data = mime.b64( fileHandle:read( “*a” ) )
io.close( fileHandle )

params.body = data

local function networkListener( event )

local response

if ( event.isError ) then
print( “Network error!”)
response = false
else
print ( "RESPONSE: " … event.response )
response = json.decode ( event.response )
end

if callback then callback ( response ) end
end

network.request( upUrl … fileName, “POST”, networkListener, params)
end


– Download file

function parse.download ( id, callback )

params.headers[“Content-Type”] = “multipart/text”
params.body = nil

local dlUrl = id[“url”]
local fileName = id[“name”]

–print ( "url: " … fileName )

local response = nil

local function networkListener( event )

if ( event.isError ) then
print( “Network error!”)
else
–print ( "RESPONSE: " … event.response )

local data = mime.unb64(event.response)

local path = system.pathForFile( fileName, system.DocumentsDirectory )
local file = io.open( path, “w+” )

if file:write( data ) then response = true end
file:close()
end

if callback then callback ( response ) end
end

network.request( dlUrl, “GET”, networkListener, params)
end [import]uid: 13632 topic_id: 31044 reply_id: 124147[/import]

Thanks ojnab,

I really appreciate the help. I see what you have done there. You are using your own encoding/decoding so are in effect ignoring the header. This means if you try access the files directly through their link in a browser they will not work, but by using your download function you do your own decoding and store the file as required.

I have tested it and its working great. I have however, noticed one thing that I thought I should mention. This method greatly adds to the file size. I tired with a 38K jpg which after upload/download became 50K. A 100K JPG became 168K and a 209K aif file became 487K. I will have a play round and see if I can figure it out, but I thought I should let you know in case it effects your code.

Antony
[import]uid: 169392 topic_id: 31044 reply_id: 124245[/import]

Yes you are right.

I worked on this 6 months ago but as far as I remember I managed to upload the files the way they describe in their api, but when downloading the files to another app the files were corrupted somehow. I think the upload was fine because I could see the images I uploaded by pasting the link in the browser. But I think there were some kind of decoding problem when downloading the files again.

I spent days getting it to work, so I ended up doing it the way I showed above even though it’s not the best solution.

Please let me know if you find a better solution. Thanks! [import]uid: 13632 topic_id: 31044 reply_id: 124314[/import]

Hey Antony

I got it working some time ago. Here is the piece of code I used.
Good luck!

[lua] local json = require (“json”)
local mime = require “mime”

local parse = {}
local headers = {}

headers[“Content-Type”] = “application/json”
headers[“X-Parse-Application-Id”] = “???”
headers[“X-Parse-REST-API-Key”] = “???”

local params = {}
params.headers = headers


– Upload file

local upUrl = “https://api.parse.com/1/files/

function parse.upload ( fileName, callback )

params.headers[“Content-Type”] = “multipart/text”

local path = system.pathForFile( fileName, system.DocumentsDirectory )
local fileHandle = io.open( path, “rb” )
local data = mime.b64( fileHandle:read( “*a” ) )
io.close( fileHandle )

params.body = data

local function networkListener( event )

local response

if ( event.isError ) then
print( “Network error!”)
response = false
else
print ( "RESPONSE: " … event.response )
response = json.decode ( event.response )
end

if callback then callback ( response ) end
end

network.request( upUrl … fileName, “POST”, networkListener, params)
end


– Download file

function parse.download ( id, callback )

params.headers[“Content-Type”] = “multipart/text”
params.body = nil

local dlUrl = id[“url”]
local fileName = id[“name”]

–print ( "url: " … fileName )

local response = nil

local function networkListener( event )

if ( event.isError ) then
print( “Network error!”)
else
–print ( "RESPONSE: " … event.response )

local data = mime.unb64(event.response)

local path = system.pathForFile( fileName, system.DocumentsDirectory )
local file = io.open( path, “w+” )

if file:write( data ) then response = true end
file:close()
end

if callback then callback ( response ) end
end

network.request( dlUrl, “GET”, networkListener, params)
end [import]uid: 13632 topic_id: 31044 reply_id: 124147[/import]

Thanks ojnab,

I really appreciate the help. I see what you have done there. You are using your own encoding/decoding so are in effect ignoring the header. This means if you try access the files directly through their link in a browser they will not work, but by using your download function you do your own decoding and store the file as required.

I have tested it and its working great. I have however, noticed one thing that I thought I should mention. This method greatly adds to the file size. I tired with a 38K jpg which after upload/download became 50K. A 100K JPG became 168K and a 209K aif file became 487K. I will have a play round and see if I can figure it out, but I thought I should let you know in case it effects your code.

Antony
[import]uid: 169392 topic_id: 31044 reply_id: 124245[/import]

Yes you are right.

I worked on this 6 months ago but as far as I remember I managed to upload the files the way they describe in their api, but when downloading the files to another app the files were corrupted somehow. I think the upload was fine because I could see the images I uploaded by pasting the link in the browser. But I think there were some kind of decoding problem when downloading the files again.

I spent days getting it to work, so I ended up doing it the way I showed above even though it’s not the best solution.

Please let me know if you find a better solution. Thanks! [import]uid: 13632 topic_id: 31044 reply_id: 124314[/import]

The issue is that network.request is badly broken and doesn’t support binary data (or as we call it in both Lua and the HTTP protocol - “data”). I debugged this for a while and wrote some tests against httpbin so I could see exactly what Corona was sending out. I submitted a bug today.

AFAICT, there is no way to upload binary data over to Parse (since it requires SSL, and thus can only use network.request, which is b0rken). You can certainly text-encode your data, as above, and then decode it when you download it, but then you won’t be storing the actual file (which is certainly an issue if you want to be able to use it directly, say as an image file, without going through your extra decoding step).

I hope they fix this, because without it, not only are we not going to be uploading files to Parse, we’re going to have the same problem with DropBox posting and who knows what else.

Alternatively, if they would just give us OpenSSL on LuaSocket, I could go back to using those APIs, which actually work quite well (and have been field-tested for a long time). network.request is just a trainwreck and it’s worse every time I look at it. [import]uid: 111481 topic_id: 31044 reply_id: 125072[/import]

@ojnab,

I’m surprised you were able to get image uploads to work to the point where you could see the image using the parse URL. In all of my testing, any upload with binary data produced a zero length upload from Corona (as verified by watching the protocol stream). Any chance you were doing this on the Windows simulator or maybe on an Android device? All of my testing has been on Mac/iOS, and I know there are some platform differences in terms of the network.request/network.download bugs. It could also be that the binary upload bug is new I guess.

There is a known bug that network.request will corrupt binary downloads on Android (reported by me, super easy fix, been open forever). Maybe that’s what you saw on download? network.download doesn’t have that particular bug (but does present a whole different set of “fun” to work around).

Bob [import]uid: 111481 topic_id: 31044 reply_id: 125082[/import]

Hi Bob,

Thanks for the explanation, you have saved me hours of hitting my head against the wall. So it seems until Corona make some changes, we are stuck with a file that increases in size every time it goes up or down.

I emailed and asked when we might also get a ZIP library as that could at least help the issue, but it seems there is no set date on that either. Heres hoping.

P.S. I use and love your facebook code. It has been a massive time saver.

[import]uid: 169392 topic_id: 31044 reply_id: 125083[/import]

Just in case anyone wants to check my work here, below is the bug report and sample code that I submitted today, which I believe proves that network.request is incapable of sending “binary” data (otherwise known as “data”). Note that I’ve also tried using network.download in all of these tests, tried various header combinations (including other c/t headers and no c/t header), and lots of other stuff.


_In attempting to upload binary files to online services via a REST interface, specifically Parse and Dropbox, they expect that the client can create a POST request where the body of said request is the binary content of the file.

The documentation for the “body” attribute of params in network.request( ) is as follows:

params.body - A string containing the HTTP body.

The documentation for the term “string” in Lua is as follows:

“Strings have the usual meaning: a sequence of characters. Lua is eight-bit clean and so strings may contain characters with any numeric value, including embedded zeros. That means that you can store any binary data into a string.”

Further, note that content/payload sent over HTTP is assumed to be binary, and that the underlying network request APIs on all Corona platforms allow for a binary request payload. In no case would this present a problem if the payload happened to be 7-bit text.

The examples below show that network.request( ) is essentially looking for a 7-bit, non-null “text” value, as opposed to processing the actual literal body provided. If a null is present in the body, network.request only processes the body up to that null. If a character with a value of greater than 127 is encountered anywhere in the body, network.request( ) transmits a zero length body.

This is not only badly broken, in that it prevents uploading of “binary” (8-bit) bodies, but also shows a profound lack of understanding of how the underlying network request mechanisms and the protocol itself work.

I would normally use the excellent LuaSocket network APIs, as I always do when I have that option, but these rest services (specifically Parse and DropBox) require an SSL connection. As there is no SSL\TLS support for LuaSocket under Corona, I have no choice but to try to use network.request. And that API is very badly broken.

Note that I am aware that there are “text based” workarounds for file uploading, specifically encoding the binary file as part of a multipart/form-data post, but that post mechanism is not supported by either of these services.

Until this is addressed, it is my opinion that it is not possible to use Parse or Dropbox, at least with binary files, from Corona.

Note: The tests are very crude, and designed to illustrate my conclusions above. I have a significant application that works with a variety of actual text and binary file, and it fails as would be expected based on these simple tests. These tests use the free httpbin auto-responding server so that you can see exactly what the server is getting from Corona._

[code]
io.output():setvbuf(‘no’)

print(“Beginning test: network.request is b0rken”)

local function fileUploadText( )

local function listener ( status )
print("fileUploadText POST response: " … status.response)
end

local params = {
headers = {
[“Content-Type”] = “text/plain”
},
body = “foo”
}

network.request(“http://httpbin.org/post”, “POST”, listener, params)

end

local function fileUploadTextWithNull( )

local function listener ( status )
print("fileUploadTextWithNull POST response: " … status.response)
end

local params = {
headers = {
[“Content-Type”] = “text/plain”
},
body = “foo\0bar”
}

network.request(“http://httpbin.org/post”, “POST”, listener, params)

end

local function fileUploadTextWithHighBit( )

local function listener ( status )
print("fileUploadTextWithHighBit POST response: " … status.response)
end

local params = {
headers = {
[“Content-Type”] = “text/plain”
},
body = “foo\128”
}

network.request(“http://httpbin.org/post”, “POST”, listener, params)

end

fileUploadText()
timer.performWithDelay(1000, fileUploadTextWithNull)
timer.performWithDelay(2000, fileUploadTextWithHighBit)
[/code] [import]uid: 111481 topic_id: 31044 reply_id: 125084[/import]

@antonyburrows - Glad you like the Facebook code! I ended up implementing a similar module for Twitter and used it in my app Pixtamatic, but haven’t released the Twitter code yet. [import]uid: 111481 topic_id: 31044 reply_id: 125085[/import]

@bob
I might remember it wrong. It’s half a year ago and I have been working on completely different things since. Wish I had you around back then to tell me it was a bug. Then I wouldn’t have wasted so much time on it. Thanks! [import]uid: 13632 topic_id: 31044 reply_id: 125094[/import]

And just in case anyone is curious, here is the issue I raised on the Parse support site:

https://www.parse.com/questions/is-there-any-way-to-upload-binary-files-other-than-with-a-binary-post-body

Unlike here, the Parse employees jump right on open issues (I got an answer literally within 3 minutes). It appears that there is no workaround for the limitations in network.request that will allow upload of binary files to Parse (which I kind of expected, just wanted to verify).

:frowning:
[import]uid: 111481 topic_id: 31044 reply_id: 125199[/import]

The issue is that network.request is badly broken and doesn’t support binary data (or as we call it in both Lua and the HTTP protocol - “data”). I debugged this for a while and wrote some tests against httpbin so I could see exactly what Corona was sending out. I submitted a bug today.

AFAICT, there is no way to upload binary data over to Parse (since it requires SSL, and thus can only use network.request, which is b0rken). You can certainly text-encode your data, as above, and then decode it when you download it, but then you won’t be storing the actual file (which is certainly an issue if you want to be able to use it directly, say as an image file, without going through your extra decoding step).

I hope they fix this, because without it, not only are we not going to be uploading files to Parse, we’re going to have the same problem with DropBox posting and who knows what else.

Alternatively, if they would just give us OpenSSL on LuaSocket, I could go back to using those APIs, which actually work quite well (and have been field-tested for a long time). network.request is just a trainwreck and it’s worse every time I look at it. [import]uid: 111481 topic_id: 31044 reply_id: 125072[/import]

@ojnab,

I’m surprised you were able to get image uploads to work to the point where you could see the image using the parse URL. In all of my testing, any upload with binary data produced a zero length upload from Corona (as verified by watching the protocol stream). Any chance you were doing this on the Windows simulator or maybe on an Android device? All of my testing has been on Mac/iOS, and I know there are some platform differences in terms of the network.request/network.download bugs. It could also be that the binary upload bug is new I guess.

There is a known bug that network.request will corrupt binary downloads on Android (reported by me, super easy fix, been open forever). Maybe that’s what you saw on download? network.download doesn’t have that particular bug (but does present a whole different set of “fun” to work around).

Bob [import]uid: 111481 topic_id: 31044 reply_id: 125082[/import]

Hi Bob,

Thanks for the explanation, you have saved me hours of hitting my head against the wall. So it seems until Corona make some changes, we are stuck with a file that increases in size every time it goes up or down.

I emailed and asked when we might also get a ZIP library as that could at least help the issue, but it seems there is no set date on that either. Heres hoping.

P.S. I use and love your facebook code. It has been a massive time saver.

[import]uid: 169392 topic_id: 31044 reply_id: 125083[/import]

Just in case anyone wants to check my work here, below is the bug report and sample code that I submitted today, which I believe proves that network.request is incapable of sending “binary” data (otherwise known as “data”). Note that I’ve also tried using network.download in all of these tests, tried various header combinations (including other c/t headers and no c/t header), and lots of other stuff.


_In attempting to upload binary files to online services via a REST interface, specifically Parse and Dropbox, they expect that the client can create a POST request where the body of said request is the binary content of the file.

The documentation for the “body” attribute of params in network.request( ) is as follows:

params.body - A string containing the HTTP body.

The documentation for the term “string” in Lua is as follows:

“Strings have the usual meaning: a sequence of characters. Lua is eight-bit clean and so strings may contain characters with any numeric value, including embedded zeros. That means that you can store any binary data into a string.”

Further, note that content/payload sent over HTTP is assumed to be binary, and that the underlying network request APIs on all Corona platforms allow for a binary request payload. In no case would this present a problem if the payload happened to be 7-bit text.

The examples below show that network.request( ) is essentially looking for a 7-bit, non-null “text” value, as opposed to processing the actual literal body provided. If a null is present in the body, network.request only processes the body up to that null. If a character with a value of greater than 127 is encountered anywhere in the body, network.request( ) transmits a zero length body.

This is not only badly broken, in that it prevents uploading of “binary” (8-bit) bodies, but also shows a profound lack of understanding of how the underlying network request mechanisms and the protocol itself work.

I would normally use the excellent LuaSocket network APIs, as I always do when I have that option, but these rest services (specifically Parse and DropBox) require an SSL connection. As there is no SSL\TLS support for LuaSocket under Corona, I have no choice but to try to use network.request. And that API is very badly broken.

Note that I am aware that there are “text based” workarounds for file uploading, specifically encoding the binary file as part of a multipart/form-data post, but that post mechanism is not supported by either of these services.

Until this is addressed, it is my opinion that it is not possible to use Parse or Dropbox, at least with binary files, from Corona.

Note: The tests are very crude, and designed to illustrate my conclusions above. I have a significant application that works with a variety of actual text and binary file, and it fails as would be expected based on these simple tests. These tests use the free httpbin auto-responding server so that you can see exactly what the server is getting from Corona._

[code]
io.output():setvbuf(‘no’)

print(“Beginning test: network.request is b0rken”)

local function fileUploadText( )

local function listener ( status )
print("fileUploadText POST response: " … status.response)
end

local params = {
headers = {
[“Content-Type”] = “text/plain”
},
body = “foo”
}

network.request(“http://httpbin.org/post”, “POST”, listener, params)

end

local function fileUploadTextWithNull( )

local function listener ( status )
print("fileUploadTextWithNull POST response: " … status.response)
end

local params = {
headers = {
[“Content-Type”] = “text/plain”
},
body = “foo\0bar”
}

network.request(“http://httpbin.org/post”, “POST”, listener, params)

end

local function fileUploadTextWithHighBit( )

local function listener ( status )
print("fileUploadTextWithHighBit POST response: " … status.response)
end

local params = {
headers = {
[“Content-Type”] = “text/plain”
},
body = “foo\128”
}

network.request(“http://httpbin.org/post”, “POST”, listener, params)

end

fileUploadText()
timer.performWithDelay(1000, fileUploadTextWithNull)
timer.performWithDelay(2000, fileUploadTextWithHighBit)
[/code] [import]uid: 111481 topic_id: 31044 reply_id: 125084[/import]

@antonyburrows - Glad you like the Facebook code! I ended up implementing a similar module for Twitter and used it in my app Pixtamatic, but haven’t released the Twitter code yet. [import]uid: 111481 topic_id: 31044 reply_id: 125085[/import]

@bob
I might remember it wrong. It’s half a year ago and I have been working on completely different things since. Wish I had you around back then to tell me it was a bug. Then I wouldn’t have wasted so much time on it. Thanks! [import]uid: 13632 topic_id: 31044 reply_id: 125094[/import]