Reading large text files

Getting Started Newbie Questions
#1

I have an incomplete idea for an app and I will need to read a large text file (maybe into a table). Any suggestions as to what type of file structure I should attempt to use eg. SQLite, Json, CVS/TVS? Each row/record will be made up of a fixed number of fields and a set separator character eg. comma, tab etc.

Name, telephone, town, postcode, date … etc

I think that I need to read the data into a table and then search it using a ‘fuzzy’ search for a user entered word and any lines that the word appears in then it will be displayed. eg. user enters MON and it may find ‘MONday’, ‘MONkey’, ‘xMONx’ and so on.

The data will be ‘read only’. Any manipulation will be done outside of the app therefore the data will be static.

I would appreciate any suggestion or pointers to tutorials,

Thanks.

#2

JSON is the simplest code wise. You open the file, read it in. Call json.decode() and you have a table of data.  It’s probably a bit more work to get it into JSON to start with.  For CSV you likely will need to write your own parser or find one someone else has already done, but its a good format to get the file into the first place.

Rob

#3

To add to what Rob suggests, I end up using http://jsonlint.com/ quite a bit in my own testing and debugging.  Pretty much every time json.decode returned an error, jsonlint said that my JSON was, in fact, not valid.

#4

JSON is the simplest code wise. You open the file, read it in. Call json.decode() and you have a table of data.  It’s probably a bit more work to get it into JSON to start with.  For CSV you likely will need to write your own parser or find one someone else has already done, but its a good format to get the file into the first place.

Rob

#5

To add to what Rob suggests, I end up using http://jsonlint.com/ quite a bit in my own testing and debugging.  Pretty much every time json.decode returned an error, jsonlint said that my JSON was, in fact, not valid.