Best way to remove illegal characters from a string

Solar2D General Questions/Discussion
#1

Hola!

I’ve got a small ‘business’ app in dev that my client will use to let people enter their data in a mySQL database (through a PHP script).

Now my challenge is removing all illegal characters from their first name and last name text fields (this mostly to prevent users from entering database commands that could screw things up for me).

I’ve tried a lot of option already using lua patterns and string.match or string.gsub, but nothing that really covers all my bases. I also have to admit that the pattern notation boggles my mind.

My main pattern problem is that name entries need to allow for spaces and hyphen/dash characters. I can run some patterns that remove punctuation and control characters, but that also removes the hyphen. Conversely, only allowing alphabetical characters does not allow for spaces or the hyphen. In a sense, a pattern that would say “replace all non-alphabetical characters with “” except for a space or hyphen” would work perfectly. But as I said, I have a hard time fully understanding the pattern lingo…

So my question:

  • Does anyone have a good method for taking all illegal characters out of a name string, but still allowing spaces and a hyphen/dash character?

If all else fails, I will probably do this character per character, check the ASCII code and append to a new empty string if the character is allowed, but I feel like there is a pattern solution that should work…

Thanks,

Thomas

#2

i know what you mean about that pattern logic, hard to grasp.

it would be really helpful if corona would provide such functions as ready API functions <hint hint>

not sure these are what you are looking for but happy to share what little i have on the formatting subject.

local function urlEncode(str) if str then str=string.gsub(str,"\n","\r\n") str=string.gsub(str,"([^%w])", function(c) return string.format("%%%02X",string.byte(c)) end) str=string.gsub(str," ","+") end return str end local function sqlEscape(str) if not str or type(str)~="string" then return str end str=str:gsub('["\'\\%z]',{['"']='\\"',['\0']='\\0',["'"]="\\'",['\\']='\\\\',}) return str end
#3

the set ([]) of all alphanumeric (%w) plus hyphen (%-) and space ( ), inverted (^), looks something like this:

s = "My-Name Is-Hyphenated" print(s:gsub("[^%w%-]",""))
#4

It sounds like you’re submitting from a Corona app to a PHP script? In which case, you should do the sanitisation in PHP rather than Corona, otherwise you’re still open to attack through packet editing. I.e. sanitising for security should always be server-side.

That in mind, you can use preg_replace() in PHP to do regex pattern replacements, but if your only purpose is to prevent SQL injection, then just using mysqli_real_escape_string() would suffice, and you should always use for user inputs that anyway…

#5

Thanks Anaqim and DaveBollinger,

Dave, that pattern magic is exactly what I was hoping someone would proffer! I just test it and it works perfectly!

#6

Hi Richard, thanks for the excellent input! I’ll try that as well!

#7

Yes, please do this on your server-side script. Those scripts can be called outside of your Corona made app and you need to secure it for all.

Rob

#8

must scrub it on both ends - allow user to see before submit, client must pass only clean data, server must entirely reject any operation with any unclean data

#9

Hi Dave,

Yes. Agreed. I’m going to implement it on multiple levels, but at the very start I already want the users to see that their input is actively being screened and sanitized, as a sort of visual deterrent to even try further meddling. So at the moment the textFields clean up their content on event.phase == “ended” or “submitted”. And then then PHP script on the server does the second step, just in case.

#10

I am trying something similar, but have not found a good solution yet.

The problem is that the German keyboard has the Dash (Gedankenstrich) to the bottom left of the numerical keyboard, and the hyphen (Bindestrich) to the bottom right. Many choose the wrong “letter” and I wanted to catch the entry of “Dash” by looking at the event.newCharacters() and replacing it if necessary

The problem is, that on the Mac I am not able to enter a Dash into the Corona code (I am using Zerobrane Studio) and have not been able to copy/paste it.

I tried comparing the string.byte() value, but the Dash and also the Euro-Sign (and I guess many others) all return 226

 local newChar = event.newCharacters local oldText = event.oldText if newChar then local theAscii = string.byte(newChar) -- native.showAlert("Letter","Newchar: "..newChar.." ascii: "..theAscii,{"OK"}) if theAscii == 226 then if oldText ~= nil then event.target.text = "-" else event.target.text = oldText.."-" end end end

Any good solutions ?

#11

Do not do this in the client!

In PHP simply use PDO and parameters for your queries.  This will completely deal with SQL injection problems and charsets issues.

#12

i know what you mean about that pattern logic, hard to grasp.

it would be really helpful if corona would provide such functions as ready API functions <hint hint>

not sure these are what you are looking for but happy to share what little i have on the formatting subject.

local function urlEncode(str) if str then str=string.gsub(str,"\n","\r\n") str=string.gsub(str,"([^%w])", function(c) return string.format("%%%02X",string.byte(c)) end) str=string.gsub(str," ","+") end return str end local function sqlEscape(str) if not str or type(str)~="string" then return str end str=str:gsub('["\'\\%z]',{['"']='\\"',['\0']='\\0',["'"]="\\'",['\\']='\\\\',}) return str end
#13

the set ([]) of all alphanumeric (%w) plus hyphen (%-) and space ( ), inverted (^), looks something like this:

s = "My-Name Is-Hyphenated" print(s:gsub("[^%w%-]",""))
#14

It sounds like you’re submitting from a Corona app to a PHP script? In which case, you should do the sanitisation in PHP rather than Corona, otherwise you’re still open to attack through packet editing. I.e. sanitising for security should always be server-side.

That in mind, you can use preg_replace() in PHP to do regex pattern replacements, but if your only purpose is to prevent SQL injection, then just using mysqli_real_escape_string() would suffice, and you should always use for user inputs that anyway…

#15

Thanks Anaqim and DaveBollinger,

Dave, that pattern magic is exactly what I was hoping someone would proffer! I just test it and it works perfectly!

#16

Hi Richard, thanks for the excellent input! I’ll try that as well!

#17

Yes, please do this on your server-side script. Those scripts can be called outside of your Corona made app and you need to secure it for all.

Rob

#18

must scrub it on both ends - allow user to see before submit, client must pass only clean data, server must entirely reject any operation with any unclean data

#19

Hi Dave,

Yes. Agreed. I’m going to implement it on multiple levels, but at the very start I already want the users to see that their input is actively being screened and sanitized, as a sort of visual deterrent to even try further meddling. So at the moment the textFields clean up their content on event.phase == “ended” or “submitted”. And then then PHP script on the server does the second step, just in case.

#20

I am trying something similar, but have not found a good solution yet.

The problem is that the German keyboard has the Dash (Gedankenstrich) to the bottom left of the numerical keyboard, and the hyphen (Bindestrich) to the bottom right. Many choose the wrong “letter” and I wanted to catch the entry of “Dash” by looking at the event.newCharacters() and replacing it if necessary

The problem is, that on the Mac I am not able to enter a Dash into the Corona code (I am using Zerobrane Studio) and have not been able to copy/paste it.

I tried comparing the string.byte() value, but the Dash and also the Euro-Sign (and I guess many others) all return 226

 local newChar = event.newCharacters local oldText = event.oldText if newChar then local theAscii = string.byte(newChar) -- native.showAlert("Letter","Newchar: "..newChar.." ascii: "..theAscii,{"OK"}) if theAscii == 226 then if oldText ~= nil then event.target.text = "-" else event.target.text = oldText.."-" end end end

Any good solutions ?