I have an app that imports a 4MByte csv file into Tiny DB which has worked for several years but the latest App Inventor version only works on relatively small files. This may be because I've changed to using 'EFile' as 'File' has a bug in process of being fixed. But I've also read that text files are now limited to 2 MByte. Is the 2 MByte limit also applied to TinyDB? Are these fundamental limits? Is there a method for importing larger files? I could split up the database into smaller units and load them separately, so long as the 2MByte limit is not also applied to Tiny DB but would like to know if there are alternative approaches.
Thank you Anke for those examples, Their basic text read operations are similar to mine so that suggests I should be able to read at least 2.5 Mbyte. In the past the largest file I have read was 3.9 MByte but I can no longer read text files that big. I'm testing various file sizes to find the limit or issue.
While searching for information I found two contradictory statements one which suggests the text file is limited by the size of the device (computer?) system memory.Variable Size Limits
That limit is for a uri entered into the address bar of a webviewer browser, not for normal download/upload functions using HTTP with the web component (GET/POST etc)
Hello Peter, that is about how much text you can put into a Text Block (or Label or TextBox), nothing specific to importing a CSV, unless you are simply importing the whole file into a Text Block (which you shouldn't be).
....and that is concerned with the method, which attaches to a URL and thus the limit is the max length of a URL - and as Taifun has pointed out right there, the method is now obsolete anyway.
It's possible that there is something wrong with the CSV file itself. If you post a representative large file and just the code snippet you use to process the file, we can help you more.
Many thanks Anke - There must be something very wrong with what I'm doing and I'm guessing it has to do with Chris' observation that I shouldn't be loading all the text into a single block. I've been looking for alternatives and must be missing something obvious. I've finished listening to Beethovens 5th and my latest 100,000 record test database still hasn't loaded. At Chris' suggestion I'm sending him an image of my cut down method and the test file that I know is not corrupted i.e. it works as a small file but extending it to a large file either takes far too long or is not working.
The problem is not reding the text, but displaying it in a Label.
It needs about 2 sec (and 5 sec for csv file of 10MB) to read. You can save the text in a variable "data":
Thanks for the explanation Chris and Tim. Anke has demonstrated that large files can be handled in about 12 seconds so there must be something wrong with my method. I've attached a cut down version of my database read write operations and a test file that I believe is not corrupted as it works when it has a small number of records i.e 15,000 but not when it is extended to 100,000. I am reading the records as a single text file using the Read File / Got Text component so that may be the problem if I understand you correctly. I'd appreciate any advice you can give, MyDB8.csv (2.6 MB)
It might go against your storage approach, but I would prefer to store the data from the csv into a list, then store the list as a whole in the tinydb. Call back the list from the tinydb and work with the list, as opposed to 100,000 tags in the tinydb. I haven't dealt with saving back to the csv because I loaded it in the assets, but all you do is save the list to the file (using a "csv table from list" block
I don't display the text in a label but do save it in a list and then transfer it to TinyDB - perhaps that is where the excessive delay is and where there may be a better method? Once TinyDB is loaded up with the database it performs quickly and well but it has always taken a long time (half an hour or more) to read the external database into TinyDB.
It could be that the excessive TinyDB load time is just too long for the phone to handle without going to sleep or shutting down?
Chris, Thanks for that suggestion - would it still allow the App user to search on tags or key words in the list? That's a critical function in my application, searching and retrieving a record has to be as simple as possible as the user will be working outdoors in a dark environment and will be focused on setting up and using the telescope rather than manipulating lists.
Would it be reasonable after the csv has been rapidly loaded to convert the list to a tinyDB format with tags? My experience with tiny db so far is that once loaded it is very fast at searching and retrieving records.
I use SQL occasionally - not all that user friendly in the environment i'm thinking of and each record only has four items so SQL might be overkill - once loaded tiny db has worked well for over the last year or two.
"Chris, Thanks for that suggestion" - did you mean me ?
These blocks search through the records in all fields for the search term, and return the first find. This could be developed to return a list of records that contain the search term for a more refined selection. All depends on how unique records and fields are. Searching on just a unique fields would return a single result.
Thanks to everyone, especially Tim for his List approach which works very well and Anke.for demonstrating that the problem was in the processing and not the data file read operation. Also apologies for getting Tim's name wrong - it was 2 am in Australia.
I've tried Tim's list approach - It's way faster than Tiny DB and fits with my App software well. Both Tiny DB and Tim's List approach retrieve records from the database quickly, Tiny DB has almost instant record retrieval, and the list approach takes about 15 seconds to retrieve records that are near the bottom of a 100,000 record list. Still perfectly acceptable for my application. Tim's approach sets up the 100,000 record list in seconds Tiny DB takes hours to assign tags to the database.
Unless my App is flawed I'd suggest the practical limit for tiny DB is about 2,000 records. It takes about 90 seconds to set up 2000 tags. The test blocks I used are copied below. I used the 100,000 record database packaged with the APK and controlled the number of records assigned tags using the text box called File Size.
Taifun.