FILE.TAIL([{lines}] would be good.
FILE.TAIL([{lines}] would be good.
I can't see an easy way of doing this currently, please let me know if I missed something. I need to load the last line of a data log file (CSV) when Annex starts up, however there doesn't appear to be a FILE option for that. I would use the FILE.READ(line... style, but I don't know the last lines number. It would be great if I could have something like FILE.TAIL([{lines}]) where it returns the last line of a file or 'lines' of a file. It's not essential, but is the way I was used to initialising the internals of my script with the last known (and probably closest) readings. I don't really want to script something to read the whole log file, just to get the last lines. I'm sure that could be done, but would be a lot faster if done internally.
Thanks,
James
Thanks,
James
-
- Posts: 142
- Joined: Mon Feb 08, 2021 10:10 pm
- Location: Scotland
- Has thanked: 44 times
- Been thanked: 50 times
Re: FILE.TAIL([{lines}] would be good.
Hi,
No - you have not missed anything.
Don't bother trying to read a large file line by line as it takes a g e s...
Assuming you are appending data to the csv file then simply either write the latest data (only) to another file or, if <512 bytes, write it to RTC RAM (BAS.RTCMEM$) and read it back instantly next time Annex starts up.
No - you have not missed anything.
Don't bother trying to read a large file line by line as it takes a g e s...
Assuming you are appending data to the csv file then simply either write the latest data (only) to another file or, if <512 bytes, write it to RTC RAM (BAS.RTCMEM$) and read it back instantly next time Annex starts up.
- cicciocb
- Site Admin
- Posts: 1899
- Joined: Mon Feb 03, 2020 1:15 pm
- Location: Toulouse
- Has thanked: 407 times
- Been thanked: 1269 times
- Contact:
Re: FILE.TAIL([{lines}] would be good.
Hi,
@bugs, Thanks, I'll look at your suggestions, I like the RTC RAM idea, but I'm not using one at the moment. I'll look into writing the last line to both the log file and another as a work-around. I was also thinking that I could get the log file size, divide it by the average line length (it varies by a few bytes so will under-shoot it), and then read lines from the calculated line number onward. As long as I under shoot the line I want, it might speed up getting to the last line. The file is going to grow to about 350k-400k, so that's why I'm trying to avoid reading all the lines.
@cicciob, Firstly, Annex RDS is really great, I'm having a lot of fun with it. Having all the libraries for the devices already loaded makes it tremendously powerful for my kind of projects, so thank you. If you're able to add a feature that would be great. Since you already offer reading from a particular line, perhaps just a count of lines would then allow me to read the last one using the current read line.
Thank You!
@bugs, Thanks, I'll look at your suggestions, I like the RTC RAM idea, but I'm not using one at the moment. I'll look into writing the last line to both the log file and another as a work-around. I was also thinking that I could get the log file size, divide it by the average line length (it varies by a few bytes so will under-shoot it), and then read lines from the calculated line number onward. As long as I under shoot the line I want, it might speed up getting to the last line. The file is going to grow to about 350k-400k, so that's why I'm trying to avoid reading all the lines.
@cicciob, Firstly, Annex RDS is really great, I'm having a lot of fun with it. Having all the libraries for the devices already loaded makes it tremendously powerful for my kind of projects, so thank you. If you're able to add a feature that would be great. Since you already offer reading from a particular line, perhaps just a count of lines would then allow me to read the last one using the current read line.
Thank You!
- Electroguard
- Posts: 836
- Joined: Mon Feb 08, 2021 6:22 pm
- Has thanked: 268 times
- Been thanked: 317 times
Re: FILE.TAIL([{lines}] would be good.
Just to clear up any confusion...I like the RTC RAM idea, but I'm not using one at the moment.
Despite the name similarity, an RTC (real time clock) module is optional, whereas RTC RAM is an area of non-volatile ESP memory which anyone can access with the Annex command:
bas.rtcmem$
(This memory is limited to 512 bytes and can be set with the corresponding command BAS.RCTMEM$ = “xxx”)
So if wanted, you do actually have 512 bytes of non-volatile memory available to use for storing any info that you may want to survive a reboot.
- cicciocb
- Site Admin
- Posts: 1899
- Joined: Mon Feb 03, 2020 1:15 pm
- Location: Toulouse
- Has thanked: 407 times
- Been thanked: 1269 times
- Contact:
Re: FILE.TAIL([{lines}] would be good.
HI,
I also think that the to use the INTERNAL RTC memory as suggested by Bugs and Electroguard is the easiest solution.
I understand that you want just check last record and add more data only if it has changed.
Take into account that reading the file line by line is very slow as the file is open and closed at each call.
This function was done considering little files (few KB) and is not adapted to big files.
Counting the number of lines can be done but I'll see how give more flexibility on the file functions.
If someone has some ideas, you are welcome.
I also think that the to use the INTERNAL RTC memory as suggested by Bugs and Electroguard is the easiest solution.
I understand that you want just check last record and add more data only if it has changed.
Take into account that reading the file line by line is very slow as the file is open and closed at each call.
This function was done considering little files (few KB) and is not adapted to big files.
Counting the number of lines can be done but I'll see how give more flexibility on the file functions.
If someone has some ideas, you are welcome.
- Electroguard
- Posts: 836
- Joined: Mon Feb 08, 2021 6:22 pm
- Has thanked: 268 times
- Been thanked: 317 times
Re: FILE.TAIL([{lines}] would be good.
A couple of things you could try...
Keep a running total of how many LINES you write to the LOG, and save bas.rtcmem$=LINES each time of writing to the log, thereby allowing you to specify the last line number of the log at any time for reading back.
Or sequence through the log file reading a line at a time and discarding the previous until reaching the end of the log file (eg: an empty line).
If you counted the LINES as you step through, then even if it takes too long to do at startup each time, at least it would give you a tool to discover the number of lines in the current log file to save using bas.rtcmem$=LINES
Keep a running total of how many LINES you write to the LOG, and save bas.rtcmem$=LINES each time of writing to the log, thereby allowing you to specify the last line number of the log at any time for reading back.
Or sequence through the log file reading a line at a time and discarding the previous until reaching the end of the log file (eg: an empty line).
If you counted the LINES as you step through, then even if it takes too long to do at startup each time, at least it would give you a tool to discover the number of lines in the current log file to save using bas.rtcmem$=LINES
- Fernando Perez
- Posts: 378
- Joined: Mon Feb 15, 2021 10:09 pm
- Location: Santander (Spain)
- Has thanked: 195 times
- Been thanked: 267 times
Re: FILE.TAIL([{lines}] would be good.
I agree with you that the best solution is to use bas.rtcmem$, but if your file is not very large and the boot time of your system is not too important, you could use something similar to this:
I am attaching a csv file of 16.633 bytes and 238 lines to experiment. It takes 8 and a half seconds.
https://myrapidq.it/public/msft.csv
By the way, Cicciocb, I have verified that if I omit the size parameter in the IObuff.ToHex $ (buff_num, [, start [, size]]) the memory overflows.
Code: [Local Link Removed for Guests]
nLines = 0
fileName$ = "/msft.csv"
t = millis
wlog "start..."
gosub tail
wlog nLines; " in "; millis - t; " millliseconds"
END
tail:
ioBuff.dim(0, file.Size(fileName$))
file.read_ioBuff(0), filename$
for i = 0 to ioBuff.len(0)
if ioBuff.toHex$(0, i, 1) = "0A" then nLines = nLines + 1
next i
ioBuff.destroy(0)
return
https://myrapidq.it/public/msft.csv
By the way, Cicciocb, I have verified that if I omit the size parameter in the IObuff.ToHex $ (buff_num, [, start [, size]]) the memory overflows.
-
- Posts: 142
- Joined: Mon Feb 08, 2021 10:10 pm
- Location: Scotland
- Has thanked: 44 times
- Been thanked: 50 times
Re: FILE.TAIL([{lines}] would be good.
Sorry - my fault for introducing the RAM confusion.
I blame "memory leak" from a previous AVR project using a DSxxxx RTC chip and its RAM.
Maybe a file.linecount function would be useful to know which line to read.
I blame "memory leak" from a previous AVR project using a DSxxxx RTC chip and its RAM.
Maybe a file.linecount function would be useful to know which line to read.
- Fernando Perez
- Posts: 378
- Joined: Mon Feb 15, 2021 10:09 pm
- Location: Santander (Spain)
- Has thanked: 195 times
- Been thanked: 267 times
Re: FILE.TAIL([{lines}] would be good.
The response time can be improved by this other procedure, but they are still excessive for files of 400 Kbytes.
If nobody proposes something better, I also think that the most suitable solution is to save the number of the last written line in RAM RTC.
Code: [Local Link Removed for Guests]
wlog "start..."
nLines = 0
fileName$ = "/msft.csv"
t = millis
gosub tail
wlog nLines; " in "; millis - t; " millliseconds"
wlog file.read$(fileName$, nLines)
END
tail:
line$ = file.read$(fileName$, 2)
nLines = int(file.Size(fileName$) / len(line$))
do
nlines = nLines - 1
loop until file.read$(fileName$, nLines) <> "_EOF_"
return