Monthly Archive

Here’s is a possible idea for the backend - the objects would be crammed into a relational database for now (unless or until there are good open source oodb’s available). A simple synchronize command would generate a file for each object (stored in YAML code or something similar). Desktop Search plugins would be indexed off of these files, and they could also be edited in Notepad and then resynched with the database.

Putting them in files would also allow storing their history in a repository like Subversion (not sure how easily extractible that history would be though, eg for field values). This would also allow use of standard merging tools for keeping the data in synch with other copies. For instance, if you have a giant database on your home machine, and then upload a chunk of it to a website (say all the objects marked as public) - then they could both be edited at the same time and kept in synch.

Eventually, I’d like to wrap all my existing files with metadata (eg by putting each file in a .zip archive along with a metadata text file (using yaml), and give it some standard extension, like .neoobj or something). These files would then sit in the folder along with all the other yaml object files, and the database would be kept in synch with their metadata also (leaving the actual binary file data on the file system). The net effect is that you would have a virtual file system sitting on top of the existing one, that is platform independent, open source, and provides metadata and history for all your files.

The Windows Explorer plugin would just be something that would expose the folder containing all these objects, and let you save files there (which would then wrap them in a metadata wrapper automatically, etc), or load them, etc. I’m not sure how this would work out yet, but it would be a way of letting legacy apps save files to your data store, instead of the old file system.

Note that if you wanted a portable version to take along with you, all you’d need is the SQLite database and a small frontend (eg the old NeoMem frontend, since it’s in C++ and makes for a small .exe). It could also be adapted for a PDA someday (which I would really like to have, to take the place of all these index cards I’m always carrying around with me). More on that later…

Here’s a rough picture of the architecture:

Developed with Graphviz (source for the drawing here).

Tags: No Tags

Here’s the latest feature/bug list: NeoMem Feature List (94kb pdf).

The next release (1.1d) will be just bug fixes (I know, not very exciting, but important). Then I’ll try and get to import/export and a few other requested features for 1.2. But I’m trying to balance my time with working on the new design as well, so some of those things might take a while. I might wind up cutting down the number of features for 1.2 in order to get import/export out. It’ll also help to have the code hosted on this site - that’s coming soon…

Note that version 1.99 is just a catchall for all future features. And 2.0 is for features that will probably best be implemented under the new architecture. And also, this still does not have a bunch of features that are scattered throughout my own notes - those will be added to the database gradually. But it incorporates everything from email, the forum, and the blog, assuming I didn’t miss anything.

And my installation of Access 97 still won’t export to HTML, despite reinstalling it, hence the pdf. I’ve been playing around with Ruby on Rails and MySQL a bit, so eventually this list will be interactive. Either that, or I’ll just put the project into Trac for the time being.

Tags: No Tags

ah well, life happens, and things get in the way - i’ve already gotten behind schedule - just a lot of distractions this last week. i’ve still been working on the database though, which feeds into ideas for the backend. and as usual, lots of other ideas occur along the way. they get jotted down on index cards, and eventually make their way into the database. and there’s a large backlog of blog entries as well.

anyway, what i’m doing is trying to cram all of my information into this (relational) database. currently the weak spots are

• allowing multiple values in a field
• composite field types
• type parsers
• relationships
• version history

i’m sure somebody, somewhere, has solved these problems before. there are some pretty advanced commercial databases out there. sure would be nice to be able to use one of those for the backend…

multiple field values - eg type=”person, performer”, date=”1789 or 1794″, date=”oct 1994 to dec 1998″, filter=”apples and oranges but not lemons”. right now i’m just storing these in the database just like that (ie text). presumably a parser could come along and store the information “properly”, though i’m not sure yet what form that will take. neomem had code to handle multiple item links, but it really needs to be expanded to be able to store arbitrary types of information with arbitrary relationships (ie not just AND).

composite field types - most databases are way too limited in their field types, so much so as to be almost useless - i’m just storing everything as text for now. but basically, all types are composite types, made up of smaller bits of information (until you get to numbers and strings). eg a person is made up of name, address, home phone, etc. an address is made up of street, city, state, zip, etc. a weight is made up of a number and a unit (eg “150 lbs”). so it would be nice to handle all of this generically, to arbitrary depths. right now neomem makes a distinction between object types and field types, but that’s pretty arbitrary (and based in part on the distinction between records and fields, which again is somewhat arbitrary). i think there are some advanced databases that let you define composite types like this, though i haven’t yet looked for any open source projects.

type parsers - ideally, each type would have its own parser, which might call subparsers. eg a NumberUnit parser would call a Number parser to interpret “150″ and a Unit parser to interpret “lbs” (the whole point being to do calculations and convert between units automatically, eventually). and dates, good grief - the number of time that wheel has been reinvented. it really should be a service of the operating system, ie pass it some text and it’ll interpret it for you. well i guess most operating systems do have such parsers but they’re nowhere near as powerful as they could be - hence, they’re basically useless. neomem’s current date parser is a step in that direction though. and all fields should also allow ranges, multiple values, and qualifiers as well, eg “jan 15, 1970 to summer 1975?”.

relationships - there are a lot of different ways of handling this, and i haven’t figured out the best way to do it yet. in some cases all you really need is to store one way information (eg .type=book), since to go the other way you can just do a search (currently how neomem handles relationships). sometimes you’d like two way storage (eg lotr.author=tolkein, tolkein.books=lotr,silmarillion) - and doing it automatically would be nice (ie you add it one place and the db adds the other side). and sometimes you’d like to attach information to the relationship (ie treat the relationship like any other object). ultimately it would be nice to have all relationships stored explicitly as objects - it would make things like displaying the entire network of objects and their relationships easier. and also for attaching “weights” to relationships, eg for building up networks of related objects.

version history - it would be nice if the database tracked the history of any field. this would include the large text fields also, allowing you to track back through the changes. and the information should be easily accessible, for viewing, editing, or feeding into plotting.

i really don’t want to reinvent any wheels though - i’m going to do some research and see what other databases are out there. hopefully there’ll be something useful…

Tags: No Tags