A faster way to read data?

Greetings,

After uploading my game Profile.txt with S3, I will download all player profiles in the game every time.

But I found that the way Corona provided was quite slow.

local function AllPlayerDataFun(Order)

       local lines = {}

       local path = system.pathForFile( “AllData”…Order…".txt", system.DocumentsDirectory )

       local file, errorString = io.open( path, “r” )

       if not file then

              – print( "File error: " … errorString )

       else

             for line in file:lines() do

                 table.insert( lines, line)

             end

             io.close( file )

              Player_UID[Order] = tostring(lines[1])

              Player_DisplayName[Order] = tostring(lines[2])

              Player_LV[Order] = tonumber(lines[3])

              Player_Head[Order] = tostring(lines[4])

              Player_HeadFrame[Order] = tonumber(lines[5])

              Player_TotalPower[Order] = tonumber(lines[6])

        end

        file = nil

        lines = nil

end

for i=1,100000 do

      AllPlayerDataFun(i)

end

I only tested 100,000 records (which is a very small value for online games), and it ran for 30 seconds!

I want to know if there is a faster way.

The second method I considered was to keep the information of all person in one file and upload it individually. 

But this practice will cause everyone’s data to be out of sync. :frowning:

Technically it’s not a method that Corona provides as much as it is one of the many methods that Lua allows. What method you should be using really depends on what kind of data you are handling.

When we are talking about save data, either local or remote, then I’m typically working with json using similar functions as described in here (https://docs.coronalabs.com/tutorial/data/jsonSaveLoad/index.html).

But if I am understanding you correctly, you are dealing with some kind of multiplayer game, where you are sending text files between players? Are you transmitting that information only once when a game starts or is this a continuous process that occurs every frame?

In any case, sending text files, if that is indeed what you are doing, wouldn’t be an optimal way, but if a single transfer takes you 0.3ms (as in 30s / 100000), then that sounds like laboratory conditions and far too optimal for real life instances.

So, long rambling cut “short”. What is it that you are actually trying to accomplish? What kind of data are you sending, how frequently and what for?

Hi XeduR,

My test just had errors. Actually it was 300s / 10000 … It seems that it is wrong to open a lot of files.

I used the json file originally, but when I downloaded it, the file was much larger than txt. After reading the document you gave, it is more suitable for SQLite.

Yes, I am dealing with multiplayer games, but this method is not use for sending text files between players.

This method is used to obtain the personal information of all players, only once in main.lua.

Since my main server is GameSparks, all player data is processed in it.

The GameSparks userId will serve as the “verification” basis for my game.

For the chat part, I used the Firebase database to store the chat history, and then extracted it. The userId was used for authentication during the extraction.

So I need to get the userId of everyone in the server before the game starts.

And I thought of another way to store a large number of files and synchronize them.

As long as I add a section to my GameSparks server, when the player is “NewPlayer”, I will send a text message to the full server person, so that everyone can add this new player and send it to Amazon S3.

Of course, this method will add a lot of server load.

So I will use the method you recommend first and later in this reply, I will tell everyone the difference between speeds.

@vb66r55  I’m really curious about your game.  Please keep us updated on its development.  How is GameSparks working out for you?

I don’t think it’s optimal to be downloading the data of every other user in the game. What happens if you get 10 million users (nice problem to have, but hey), they’d be sitting there all day waiting for the app to start. Shouldn’t you just request data as you need it in manageable chunks? 

It’s exactly as Nick said. You should not, under any circumstance, just download a large chunk of data that 99.99% of players have no need for.

A better method could be having a database with every player’s necessary information that is accessed on a per use case. It sounds like you have some type of authoritative server model going on, so you could simply let the server send everyone a JSON string with the necessary information when the game starts. This way you’d send only a fraction of the data so it’d be faster to process and it wouldn’t bankrupt your users who are playing on some data plan.

@Sporkfin,

Since I’m busy solving these problems first, I will explain the match with GameSparks later.

@nick @XeduR

I also understand this very well, but considering what might happen.

  1. When the player enters the ID of his friend, he cannot find it.

  2. Only part of the team (guild) will appear at a time, which will cause some teams to not be known.

These methods, I can use as XeduR said.

Get that important piece of information directly at my GameSparks.

But the problem comes, I can’t afford the “enterprise” level at the moment, so I need to make certain restrictions to reduce the load on the server.

I am using Corona’s sqlite3 and testing the feasibility of 1 million records.

After all, if it exceeds 1 million, it means that I have made a huge gains and the server will eventually be upgraded to the enterprise level.

After testing, I will explain here whether SQLite can support 1 million records.

Surely they have limits not just on api calls but bandwidth too? One call downloading 50MB is a lot worse for them than 10 calls requesting data for a specific user.

It’s chicken and egg really, your game is not going to have tens of thousands of users to start with so I suspect you won’t need to get round API call limits until the point where the number of users can support upgrading anyway. The solutions should be scalable.

SGS is the guy to ask, he’s built very successful games in Corona that have a multiplayer element involving lots of data.

@nick

Yes,they have limit 25MB,so instead of using their download service, I use AmazonS3.

What I think really affects is “response time”, which is the problem that really affects the additional information.

They have a fair limit of 1,000 one call for “standard” tier users ($ 300 USD per month).

This means that you must have <1000 lines of data at a time and limit call 1000 times in one second.

Overtime problems also occur very often, which can be said to be a very easy traffic jam number, which will make everyone have to wait in line.

Now I use SQLite. When I test add 1 million records, the system memory will be so high that it becomes unusable. It will take 75 seconds to return to normal.

Correct, when I just conducted the test, the information was not written, haha.

Writing is quite time consuming during testing, but reading data is the opposite.

The process of reading data will not affect the performance of the game at all.

Reading tens of thousands of data is perfectly fine.

I wrote in the GameSpark server any measures to prevent theft and affect the fairness of the game.

But regarding this SQLite (Database).

It is shared by everyone and updated and uploaded by others every time. Since he does not operate in my server, what can I do to prevent him from being modified?

I saw SGS in the forum, he has strong technology and achievements.

Hi everyone,

After referring to every comments, I solved this problem.

Work flow is ↓

1.local GameDataArray = {}

  1. Download “GameData.db”, search for the data, and save the data in the GameDataArray.

  2. Use the GameSparks server for identity verification, and send personal data from the server to the client.

  3. Use table.indexOf (GameDataArray, MyUserID) to query.

4.If it is Nil then WriteNewDataForDataBase.(DataBase:exec(NewData))

  1. If it exists UpdateDataBaseData .

6.Search the data again and update the data in GameDataArray.

  1. Upload “GameData.db” to AmazonS3.

In this way, it will be able to synchronize and update the people’s information in the full server, and can also run normally and quickly under the huge data.

I’m quite sure you still end up paying for uploading to Amazon S3. The data transfer to Amazon might not cost anything, but I’d double check if they charge for data storage (where you kinda play for whatever you upload there) and if they charge for PUT and POST requests.

They do have free tier and I’m not sure what it covers, but I wouldn’t expect uploads to S3 to be free.

You are right. I read 0.00USD before, maybe I read it wrong.

“Request” It needs 0.004 per GB. :stuck_out_tongue:

So I deleted my paragraph above.

Hi XeduR,

I just went to see the pricing, and the data is really free for S3. And the money to pay is “Request”.

@sporkfin,

I’m sorry for the late reply.

Regarding my game, I am still in the process of making it. At the end of the month, I will start shooting promotional videos, and I will also put it here, but it is not convenient to reveal too much game content now.

As for how GameSparks can help me work, I must first explain that there are some restrictions on the “standard level”, such as chat and multiplayer games. These are not available.

So after considering money, I used【GameSparks】【Photon】【FireBase】【AmazonS3】to complete my game together. The first two have official support from Corona, and the last two have plugins written by community partners in the market.

GameSparks is the main server, and although Photon and FireBase also have identity verification functions, I don’t use them. I only use GameSparks userID for unified verification.

In GameSparks, you can set multiple currencies to protect your “paying gems” from malicious modification.

For the current popular lottery mode, it is meaningless no matter how high the player changes the client’s gem.

Because I wrote in GameSparks’ Cloud Code, I checked the number of gems during the draw, and updated the prizes on the Server to avoid being modified.

Then GameSparks’s messaging function, I like this very much. I can send messages to people online when certain conditions are triggered.

Example: sporkfin defeated vb66r55 in this battle.

Multiplayer battles can use Photon, and when winning, you can use GameSparks increase your sense of honor as mentioned above.

Then there is the function of sending gifts, which is now a must-have feature in multiplayer games. I can interact with viewers on FaceBook or Twitter and send gifts to all people online.

Although I still use many other features, this article seems too long.

All in all, the functions of multiplayer online games can be achieved with these services.

Thanks @vb66r55 - that is useful info, I appreciate it.  I can’t wait try your game, let us know when you release it.

Yes, I will.

Please wait for my good news. :smiley: