I have a object that contains radium child objects much like Directories/Folders
The objects contain pictures and data
And these objects use its data for screen display purposes
Their are a multitude of this data I want stored in a single file
So obviously I can use TFile and do a update to the file that is made with a 2nd TFile
I do not see any other way but to rebuild the file to add into the file another picture or pictures.
Their is a delay using a TFile with so many pictures transferred separately between the 2 TFiles, ones loading and another is saving and the searching through the object to keep data order so the loading of data is fast. I can use a separate thread while a file update is in progress so my App can do other things.
A possibility is a recording system of instructions to my object and radium child objects that is held until the File updating process can accommodate the data modifications.
I know the windows File system runs a very small Buffer under the read and write
Does Delphi offer a buffered TFile object of any type and the buffering operates a separate thread and especially for writing Files some how, so data is stacked in RAM until data loading has paused?
Another option is use a SSD card I guess to get data transfer speed
In the future I would like the file system accessed over the internet as most use of the file system will be reading data only.
The question is how should I be looking at this problem
Hi Lex, my first question would be why you want to store it all in a single file. Using individual files for each object, or storing them in a relational or object database would simplify things such as extending the size of the file, re-writing an object with a larger size, deleting objects, diagnosing and fixing issues with the saved objects. If a single file is a definite requirement then you could look at a memory mapped file, where the reading/writing threads are accessing the shared memory and the OS handles the filesystem operations.
In any case you still need to solve the multi-threaded access, which would be simplest having a single thread write at a time but allowing multiple readers if there is currently no write operation. That depends on how frequently you are writing and the size of the objects. You can do that using a TMultiReadExclusiveWriteSynchronizer.
If you do need multiple writers at the same time, writing different objects and only blocking if two write the same object, then you would need to implement some kind of locking list (itself being locked) that tracks which objects are being written to (e.g. a TDictionary<objectID, IReadWriteSync> where the values are instances of TMultiReadExclusiveWriteSynchronizer. You could hide that within a class with methods like BeginWrite(objectID), EdnWrite(objectID), BeginRead(…) …
As for a buffered file you can look at TBufferedFileStream.
Jarrod I’m just trying to bounce idea’s
I did not know of System.SysUtils.TMultiReadExclusiveWriteSynchronizer
Thanks, that can be useful
So with a good file naming system and all the files in a single folder I simply break up a main file that the full rebuilding of a main file is not necessary I was not thinking that way but its very possible at the same time. I was thinking a single file as much more portable.
I have 255 characters to a file name correct?
2 options immediately come to mind (caveat: I have no idea what a radium child object is):
zip file - all images can be stored as individual files, you can control the directory structure within the zip, as well as naming conventions, etc. This should map to your existing radium folder metaphor
sqlite - images can be stored in sqlite blob fields
In both cases, all the read / write is handled for you and it’s all stored in a single file by default. No file rebuild, etc, necessary.
Its an idea but not the direction I want as zip requires compression time and is more difficult than a single file. Jpeg pictures do not compress much under Zip anyway.
May be I’m looking for a folder to looks like a file from the computer users point of view
If you know what I mean.
Thanks for your idea’s as that is why I put the post up for to get idea’s
I too considered mentioning using a file format like zip (it has a store only/zero compression option) or tar. These file formats support adding/appending files without rewriting the entire archive file but the libraries generally don’t support that and end up copying + appending, or rewriting the entire file (I haven’t checked the TZipFile class included with Delphi).
Aaron mentioned sqlite. There are a few databases that support local single file DBs (you don’t need a full blown DB server or a full install or service running but may need to deploy a DLL with your app).
As for max filename size, yes, unfortunately you are limited to 260 chars for the entire folder path + filename. The limit can be increased on Win10+ with a registry change but I’ve never played with that (not sure if standard on Win11).
From the gist of it, it seems that if you do play around with the registry, you also need to add an application manifest to inform Windows that the application can handle long file paths/names.
That is a good idea for transport I guess is to zip or tar a group of files for transport
and Zip can offer separation of files with zero compression to save my code management of each files data between the files.
Many be that’s the real answer I have been looking for, good thinking - thanks guys
Nicholas Ring
I have been thinking on that and I think is part of trail and error I guess
A separate registry file holding keys is one thing or to read the file names in the folder
Their are other issues a file registry repair… many issues to consider
I have been thinking is write an updated file beside then delete the old one and rename the new one to the old name to replace it. Its better way for when a power loss happens that data is maintained and recovery is easier.
Thanks all of you - its been good to bounce idea’s off
I believe I have a good direction with this project now
Its no much better to ask these questions first before wasting a lot of time going no where